github/kubernetes
4
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
c40a668ae4
kubernetes/cluster/gce/gci
History
Kubernetes Submit Queue 6dfe5c49f6 Merge pull request #38865 from vwfs/ext4_no_lazy_init 
Automatic merge from submit-queue

Enable lazy initialization of ext3/ext4 filesystems

**What this PR does / why we need it**: It enables lazy inode table and journal initialization in ext3 and ext4.

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #30752, fixes #30240

**Release note**:
```release-note
Enable lazy inode table and journal initialization for ext3 and ext4
```

**Special notes for your reviewer**:
This PR removes the extended options to mkfs.ext3/mkfs.ext4, so that the defaults (enabled) for lazy initialization are used.

These extended options come from a script that was historically located at */usr/share/google/safe_format_and_mount* and later ported to GO so this dependency to the script could be removed. After some search, I found the original script here: https://github.com/GoogleCloudPlatform/compute-image-packages/blob/legacy/google-startup-scripts/usr/share/google/safe_format_and_mount

Checking the history of this script, I found the commit [Disable lazy init of inode table and journal.](4d7346f7f5). This one introduces the extended flags with this description:
```
Now that discard with guaranteed zeroing is supported by PD,
initializing them is really fast and prevents perf from being affected
when the filesystem is first mounted.
```

The problem is, that this is not true for all cloud providers and all disk types, e.g. Azure and AWS. I only tested with magnetic disks on Azure and AWS, so maybe it's different for SSDs on these cloud providers. The result is that this performance optimization dramatically increases the time needed to format a disk in such cases.

When mkfs.ext4 is told to not lazily initialize the inode tables and the check for guaranteed zeroing on discard fails, it falls back to a very naive implementation that simply loops and writes zeroed buffers to the disk. Performance on this highly depends on free memory and also uses up all this free memory for write caching, reducing performance of everything else in the system. 

As of https://github.com/kubernetes/kubernetes/issues/30752, there is also something inside kubelet that somehow degrades performance of all this. It's however not exactly known what it is but I'd assume it has something to do with cgroups throttling IO or memory. 

I checked the kernel code for lazy inode table initialization. The nice thing is, that the kernel also does the guaranteed zeroing on discard check. If it is guaranteed, the kernel uses discard for the lazy initialization, which should finish in a just few seconds. If it is not guaranteed, it falls back to using *bio*s, which does not require the use of the write cache. The result is, that free memory is not required and not touched, thus performance is maxed and the system does not suffer.

As the original reason for disabling lazy init was a performance optimization and the kernel already does this optimization by default (and in a much better way), I'd suggest to completely remove these flags and rely on the kernel to do it in the best way.
2017-01-18 09:09:52 -08:00
..
mounter Remove all MAINTAINER statements in the codebase as they aren't very useful and now deprecated 2016-12-17 20:34:10 +02:00
configure-helper.sh Merge pull request #38865 from vwfs/ext4_no_lazy_init 2017-01-18 09:09:52 -08:00
configure.sh Generate a kubelet CA and kube-apiserver cert-pair for kubelet auth. 2017-01-03 14:30:45 -08:00
health-monitor.sh Merge pull request #39691 from yujuhong/bump_timeout 2017-01-10 21:25:16 -08:00
helper.sh Ensure the GCI metadata files do not have whitespace at the end 2016-12-13 13:41:54 -08:00
master-helper.sh Generate a kubelet CA and kube-apiserver cert-pair for kubelet auth. 2017-01-03 14:30:45 -08:00
master.yaml Add a bare-bones level of indirection for GCI mounter 2016-10-24 05:50:24 -07:00
node-helper.sh Splitting OS_DISTRIBUTION into NODE_OS_DISTRIBUTION and 2016-07-15 12:02:31 -07:00
node.yaml Add a bare-bones level of indirection for GCI mounter 2016-10-24 05:50:24 -07:00
README.md Update Container-VM Image product name in docs 2016-09-09 10:08:56 -07:00

README.md

Container-VM Image

Container-VM Image is a container-optimized OS image for the Google Cloud Platform (GCP). It is primarily for running Google services on GCP. Unlike the open preview version of container-vm, the new Container-VM Image is based on the open source ChromiumOS project, allowing us greater control over the build management, security compliance, and customizations for GCP.

Analytics