github/kubernetes
4
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
90bc19959d
kubernetes/hack
History
Kubernetes Submit Queue 63139f937c Merge pull request #30787 from jbeda/rsync 
Automatic merge from submit-queue

Speed up dockerized builds

This PR speeds up dockerized builds.  First, we make sure that we are as incremental as possible.  The bigger change is that now we use rsync to move sources into the container and get data back out.

To do yet:
* [x] Add a random password to rsync.  This is 128bit MD4, but it is better than nothing.
* [x] Lock down rsync to only come from the host.
* [x] Deal with remote docker engines -- this should be necessary for docker-machine on the mac.
* [x] Allow users to specify the port for the rsync daemon.  Perhaps randomize this or let docker pick an ephemeral port and detect the port?
* [x] Copy back generated files so that users can check them in.  This is done for `zz_generated.*` files generated by `make generated_files` 
  * [x] This should include generated proto files so that we can remove the hack-o-rama that is `hack/hack/update-*-dockerized.sh` 
* [x] Start "versioning" the build container and the data container so that the CI system doesn't have to be manually kicked.
* [x] Get some benchmarks to qualify how much faster.

This replaces #28518 and is related to #30600.

cc @thockin @spxtr @david-mcmahon @MHBauer 

Benchmarks by running `make clean ; sync ; time bash -xc 'time build/make-build-image.sh ; time sync ; time build/run.sh make ; time sync; time build/run.sh make'` on a GCE n1-standard-8 with PD-SSD.

| setup | build image | sync | first build | sync | second build | total |
|-------|-------------|----- |----------|------|--------------|------|
| baseline | 0m11.420s | 0m0.812s | 7m2.353s | 0m42.380s | 7m8.381s | 15m5.348s |
| this pr | 0m10.977s | 0m15.168s | 7m31.096s | 1m55.692s | 0m16.514s | 10m9.449s |
2016-10-05 19:51:09 -07:00
..
boilerplate Remove "All rights reserved" from all the headers. 2016-06-29 17:47:36 -07:00
cmd/teststale Remove "All rights reserved" from all the headers. 2016-06-29 17:47:36 -07:00
e2e-internal [Federation] Sleep for 2m (arbitrarily chosen) to let the underlying clusters to clean up all the resources they are holding. 2016-09-29 13:46:36 -07:00
gen-swagger-doc change the relative links to definition in operations.html to satisfy the new path set in the kubernetes.io 2016-07-29 13:29:34 -07:00
jenkins Remove unused bash variables 2016-10-03 19:42:22 -07:00
lib Install test subpackages too 2016-10-03 19:42:21 -07:00
make-rules Merge pull request #34020 from juanvallejo/jvallejo/allow-empty-annotation-values 2016-10-05 11:48:02 -07:00
testdata add a test to test-cmd.sh for apply -f with label selector 2016-09-30 00:42:35 -07:00
verify-flags Merge pull request #31467 from vmware/fix-vsphere-cloud-provider.kerneltime 2016-10-05 19:12:17 -07:00
.linted_packages Make genericapiserver handler chain customizable 2016-10-05 10:32:36 -04:00
autogenerated_placeholder.txt Revamp doc generation 2016-06-08 17:22:12 -07:00
benchmark-go.sh Use make as the main build tool 2016-07-12 21:52:00 -07:00
benchmark-integration.sh Use make as the main build tool 2016-07-12 21:52:00 -07:00
build-cross.sh Make releases work 2016-07-12 21:52:54 -07:00
build-go.sh Use make as the main build tool 2016-07-12 21:52:00 -07:00
build-ui.sh Cleanup non-rest apiserver handlers 2016-09-15 13:22:45 +02:00
cherry_pick_pull.sh hack: Update cherry-pick script to include original PR subject 2016-10-04 13:45:22 -07:00
dev-build-and-push.sh Remove "All rights reserved" from all the headers. 2016-06-29 17:47:36 -07:00
dev-build-and-up.sh Remove "All rights reserved" from all the headers. 2016-06-29 17:47:36 -07:00
dev-push-hyperkube.sh Handle spaces in some hack/ scripts. Cleanup after hyperkube build. 2016-08-15 21:13:13 +00:00
e2e-node-test.sh Use make as the main build tool 2016-07-12 21:52:00 -07:00
e2e.go e2e.go / kops: Add --kops-kubernetes-version 2016-10-04 14:01:31 -07:00
federated-ginkgo-e2e.sh Remove "All rights reserved" from all the headers. 2016-06-29 17:47:36 -07:00
generate-docs.sh Use make as the main build tool 2016-07-12 21:52:00 -07:00
get-build.sh Merge pull request #26755 from david-mcmahon/fix-headers 2016-06-29 18:46:07 -07:00
ginkgo-e2e.sh Get network name via e2e environment. 2016-08-26 13:31:08 -07:00
godep-save.sh Make script more friendly 2016-09-22 13:41:27 -07:00
grab-profiles.sh Remove "All rights reserved" from all the headers. 2016-06-29 17:47:36 -07:00
install-etcd.sh Remove "All rights reserved" from all the headers. 2016-06-29 17:47:36 -07:00
list-feature-tests.sh Remove "All rights reserved" from all the headers. 2016-06-29 17:47:36 -07:00
local-up-cluster.sh Merge pull request #30787 from jbeda/rsync 2016-10-05 19:51:09 -07:00
lookup_pull.py Remove "All rights reserved" from all the headers. 2016-06-29 17:47:36 -07:00
OWNERS Add jbeda to OWNERS for build, cluster, hack 2016-09-27 14:53:16 -07:00
run-in-gopath.sh Allow make to run from outside GOPATH 2016-07-15 08:42:12 -07:00
test-cmd.sh fix hack/test-cmd 2016-08-02 10:27:29 -04:00
test-go.sh Use make as the main build tool 2016-07-12 21:52:00 -07:00
test-integration.sh choose a particular directory test-integration 2016-08-26 12:33:06 -04:00
test-update-storage-objects.sh stick to 2.2.1 etcd 2016-09-09 16:35:05 -07:00
update_owners.py Don't assign anything to Sami anymore 2016-09-01 22:52:13 -07:00
update-all.sh hack: run openapi-spec as part of update-all.sh 2016-10-04 16:43:44 +02:00
update-api-reference-docs.sh update generation scripts to share API group version constants 2016-09-22 13:30:41 -04:00
update-bindata.sh Make update-bindata be incremental build friendly 2016-10-03 19:42:20 -07:00
update-codecgen.sh added copy.sh which cuts the client repo; made necessary changes to scripts in 2016-08-09 14:07:53 -07:00
update-codegen.sh Daemonset in federated clientset 2016-10-05 19:58:49 +02:00
update-federation-api-reference-docs.sh Adding a script to update federation API reference docs 2016-08-31 13:21:42 -07:00
update-federation-generated-swagger-docs.sh Adding a script to update federation API reference docs 2016-08-31 13:21:42 -07:00
update-federation-swagger-spec.sh Adding update-federation-generated-swagger-docs to generate comment funcs for federation gvs 2016-08-30 14:15:21 -07:00
update-generated-docs.sh Use make as the main build tool 2016-07-12 21:52:00 -07:00
update-generated-protobuf-dockerized.sh Fix subtle build breakage 2016-08-12 21:20:38 -07:00
update-generated-protobuf.sh Make update-generated-*.sh not hack around with docker images 2016-10-03 19:42:21 -07:00
update-generated-runtime-dockerized.sh Make update-generated-runtime work 2016-07-12 21:52:54 -07:00
update-generated-runtime.sh Make update-generated-*.sh not hack around with docker images 2016-10-03 19:42:21 -07:00
update-generated-swagger-docs.sh update generation scripts to share API group version constants 2016-09-22 13:30:41 -04:00
update-godep-licenses.sh Add a bash version check and some suggested updates for OSX. 2016-09-29 13:38:22 -07:00
update-gofmt.sh Merge pull request #31547 from mbohlool/fix2 2016-09-14 05:35:51 -07:00
update-munge-docs.sh Use make as the main build tool 2016-07-12 21:52:00 -07:00
update-openapi-spec.sh Generate and verify openapi specs in source tree at api/openapi-spec 2016-09-29 17:03:31 -07:00
update-swagger-spec.sh update generation scripts to share API group version constants 2016-09-22 13:30:41 -04:00
verify-all.sh Use make as the main build tool 2016-07-12 21:52:00 -07:00
verify-api-reference-docs.sh Remove "All rights reserved" from all the headers. 2016-06-29 17:47:36 -07:00
verify-boilerplate.sh Remove "All rights reserved" from all the headers. 2016-06-29 17:47:36 -07:00
verify-codecgen.sh Remove "All rights reserved" from all the headers. 2016-06-29 17:47:36 -07:00
verify-codegen.sh Remove "All rights reserved" from all the headers. 2016-06-29 17:47:36 -07:00
verify-description.sh Use make as the main build tool 2016-07-12 21:52:00 -07:00
verify-flags-underscore.py ignore staging in munge scripts 2016-08-24 13:09:13 -07:00
verify-generated-docs.sh Use make as the main build tool 2016-07-12 21:52:00 -07:00
verify-generated-protobuf.sh Use rsync to get source into build container 2016-10-03 19:42:20 -07:00
verify-generated-runtime.sh Proposal for client-server container runtime 2016-07-12 09:17:08 +08:00
verify-generated-swagger-docs.sh Use rsync to get source into build container 2016-10-03 19:42:20 -07:00
verify-godep-licenses.sh Remove "All rights reserved" from all the headers. 2016-06-29 17:47:36 -07:00
verify-godeps.sh make verify godeps help you fix your problems 2016-09-21 18:08:41 -07:00
verify-gofmt.sh ignore staging in munge scripts 2016-08-24 13:09:13 -07:00
verify-golint.sh fix hack/verify-golint.sh to work outside of GOPATH 2016-09-08 11:07:49 -04:00
verify-govet.sh Use make as the main build tool 2016-07-12 21:52:00 -07:00
verify-import-boss.sh Use make as the main build tool 2016-07-12 21:52:00 -07:00
verify-linkcheck.sh Use make as the main build tool 2016-07-12 21:52:00 -07:00
verify-munge-docs.sh Add missing mungedoc run to hack/verify-munge-docs.sh. 2016-07-13 13:39:20 -07:00
verify-openapi-spec.sh Generate and verify openapi specs in source tree at api/openapi-spec 2016-09-29 17:03:31 -07:00
verify-swagger-spec.sh Use make as the main build tool 2016-07-12 21:52:00 -07:00
verify-symbols.sh Use make as the main build tool 2016-07-12 21:52:00 -07:00
verify-test-images.sh Remove "All rights reserved" from all the headers. 2016-06-29 17:47:36 -07:00