Commit Graph

887 Commits

Author SHA1 Message Date
David Porter
893f5fd4f0 Promote kubelet graceful node shutdown to beta
- Change the feature gate from alpha to beta and enable it by default

- Update a few of the unit tests due to feature gate being enabled by
  default

- Small refactor in `nodeshutdown_manager` which adds `featureEnabled`
  function (which checks that feature gate and that
  `kubeletConfig.ShutdownGracePeriod > 0`).

- Use `featureEnabled()` to exit early from shutdown manager in the case
  that the feature is disabled

- Update kubelet config defaulting to be explicit that
  `ShutdownGracePeriod` and `ShutdownGracePeriodCriticalPods` default to
  zero and update the godoc comments.

- Update defaults and add featureGate tag in api config godoc.

With this feature now in beta and the feature gate enabled by default,
to enable graceful shutdown all that will be required is to configure
`ShutdownGracePeriod` and `ShutdownGracePeriodCriticalPods` in the
kubelet config. If not configured, they will be defaulted to zero, and
graceful shutdown will effectively be disabled.
2021-03-05 15:21:37 -08:00
Matthias Bertschy
431e6a7044 Move readinessManager updates handling to kubelet 2021-03-05 07:02:25 +01:00
Geonju Kim
b4b7cea413 kubelet_test: Add TestHandlePodRemovesWhenSourcesAreReady 2021-02-26 06:34:27 +09:00
Geonju Kim
256447a349 kubelet_test: Fix TestHandlePodCleanups 2021-02-26 06:34:17 +09:00
Geonju Kim
fc4a29da2c kubelet: Make the test fail if (*FakeRuntime).Assert fails 2021-02-26 06:31:54 +09:00
Kubernetes Prow Robot
17c3ee8708
Merge pull request #98742 from gjkim42/sync-until-terminate-containers
kubelet: Sync completed pods until their containers have been terminated
2021-02-24 15:29:26 -08:00
Geonju Kim
5e752968c3 Simulate KillPod in TestSyncPodsDeletesWhenSourcesAreReadyPerQOS 2021-02-14 07:17:48 +09:00
Geonju Kim
b451c15bf7 kubelet: Fix race when KillPod followed by IsPodPendingTermination
Ensures the pod to be pending termination or be killed, after
(*podKillerWithChannel).KillPod has been returned, by limiting
one request per pod in (*podKillerWithChannel).KillPod.
2021-02-14 07:16:49 +09:00
Ryan Phillips
ee8ea1b2c1 kubelet_test: fixes race in TestSyncPodsDeletesWhenSourcesAreReadyPerQOS 2021-02-10 09:36:32 -06:00
Kubernetes Prow Robot
45d9a13b94
Merge pull request #96451 from ping035627/k8s-201112
Extract the const for ContainerStateReason
2021-02-09 10:25:00 -08:00
Geonju Kim
321ca8af52 kubelet: Sync completed pods until their containers have been terminated 2021-02-06 14:06:50 +09:00
Ryan Phillips
f918e11e3a register all pending pod deletions and check for kill
do not delete the cgroup from a pod when it is being killed
2021-02-04 11:45:42 -06:00
PingWang
4103ff490f Extract the const for ContainerStateReason
Signed-off-by: PingWang <wang.ping5@zte.com.cn>

update fmt

Signed-off-by: PingWang <wang.ping5@zte.com.cn>

update test

Signed-off-by: PingWang <wang.ping5@zte.com.cn>
2021-02-04 08:52:13 +08:00
Kubernetes Prow Robot
4e93dbcd0d
Merge pull request #94087 from derekwaynecarr/node-sync-once
kubelet waits for node lister to sync at least once
2021-01-12 15:06:35 -08:00
Derek Carr
acb43c7c4a Rework hostfs metrics
Ephemeral storage usage should be calculated by the metrics code,
not the eviction code.
2020-12-03 13:04:25 -07:00
Joel Smith
39a11744ce Partially revert "Include pod /etc/hosts in ephemeral storage calculation for eviction"
This reverts (most of) commit f34b586d01.
2020-12-03 04:47:16 -07:00
Kubernetes Prow Robot
d21815cb4e
Merge pull request #95569 from oomichi/remove-kubelet-util
Move dirExists() to kubelet_test
2020-11-06 11:28:51 -08:00
Ali
bfdeda58b7 Delete framework/v1alpha1 folder and change remaining import paths 2020-10-23 13:16:13 +11:00
David Eads
ff7d1444f0 kubelet container status calculation doesn't handle suddenly missing data properly 2020-10-15 12:26:16 -04:00
Kenichi Omichi
c0795782e0 Move dirExists() to kubelet_test
dirExists() is called from kubelet_test only.
This moves the function to kubelet_test for cleanup.
2020-10-14 17:50:00 +00:00
Srini Brahmaroutu
fbe5daed73 Change code to use staging/k8s.io/mount-utils 2020-09-16 21:51:24 -07:00
David Dymko
cda0070f28 fix golint for pkg/volume/azure_dd 2020-09-05 09:44:27 -04:00
Derek Carr
752135242e WIP: node sync at least once 2020-09-01 12:17:26 -04:00
Kubernetes Prow Robot
fe1aeff2d2
Merge pull request #92013 from MHBauer/dockershim-error-test
basic regression test of runDockershim
2020-08-27 17:55:55 -07:00
Jordan Liggitt
b181c76cbd Deflake TestUpdateNodeStatusWithLease - guard cached machineInfo 2020-08-05 10:00:36 -04:00
Jordan Liggitt
124a5ddf72 Fix int->string casts 2020-07-24 16:23:12 -04:00
Kubernetes Prow Robot
8398bc3b53
Merge pull request #92916 from joelsmith/count-etc-hosts
Include pod /etc/hosts in ephemeral storage calculation for eviction
2020-07-12 06:59:36 -07:00
Kubernetes Prow Robot
93e76f5081
Merge pull request #92442 from tedyu/grace-period-with-map
Respect grace period when removing mirror pod
2020-07-10 17:49:23 -07:00
Kubernetes Prow Robot
1e3eeba9fa
Merge pull request #91577 from knabben/kubelet-bootstrap
kubelet: remove the --bootstrap-checkpoint-path feature
2020-07-09 00:03:41 -07:00
Ted Yu
a76a959294 Respect grace period when removing mirror pod
Signed-off-by: Ted Yu <yuzhihong@gmail.com>
2020-07-08 13:38:24 -07:00
Joel Smith
f34b586d01 Include pod /etc/hosts in ephemeral storage calculation for eviction 2020-07-08 12:58:11 -06:00
Sergey Kanzhelev
ee53488f19 fix golint issues in pkg/kubelet/container 2020-06-19 15:48:08 +00:00
Morgan Bauer
cb4b67a886
basic regression test of runDockershim
- added basic regression test to ensure an error is raised in the
   case of an unconfigured runtime, and the case of asking for a docker
   runtime when compiled dockerless

Signed-off-by: Morgan Bauer <mbauer@us.ibm.com>
2020-06-16 10:08:22 -07:00
Amim Knabben
0ed41c3f10 Deprecating --bootstrap-checkpoint-path flag 2020-06-09 15:27:01 -04:00
David Eads
4da0e64bc1 reduce race risk in kubelet for missing KUBERNETES_SERVICE_HOST 2020-05-29 17:11:19 -04:00
Abdullah Gharaibeh
d6522e0e74 rename framework pkg with schedulerframework for all instances under pkg/kubelet 2020-04-14 14:24:07 -04:00
Abdullah Gharaibeh
bed9b2f23b Cleanup obsolete NodeInfo methods 2020-04-12 18:13:46 -04:00
louisgong
7f5076d8ee slim down some lister expansions 2019-12-07 08:27:06 +08:00
Travis Rhoden
0c5c3d8bb9
Remove pkg/util/mount (moved out of tree)
This patch removes pkg/util/mount completely, and replaces it with the
mount package now located at k8s.io/utils/mount. The code found at
k8s.io/utils/mount was moved there from pkg/util/mount, so the code is
identical, just no longer in-tree to k/k.
2019-11-15 08:29:12 -07:00
Kubernetes Prow Robot
a08b09d52f
Merge pull request #84279 from matthyx/kuberuntime-startupprobe
Add startupProbe result handling to kuberuntime
2019-11-13 13:01:53 -08:00
Matthias Bertschy
66595d54a0 Add startupProbe result handling to kuberuntime 2019-11-13 08:12:54 +01:00
Kubernetes Prow Robot
897ce3073c
Merge pull request #84533 from davidz627/fix/deprecatedPath
Remove plugin watching of deprecated directory and CSI v0 support in accordance with deprecation policy
2019-11-12 04:48:20 -08:00
David Zhu
802fe12803 Remove plugin watching of deprecated directory {kubelet_root_dir}/plugins and support for CSI V0 in accordance with deprecation announcement in https://v1-13.docs.kubernetes.io/docs/setup/release/notes/ 2019-11-11 11:42:58 -08:00
Travis Rhoden
1fd8921546
Move mount/fake.go to mount/fake_mount.go
This patch moves fake.go to mount_fake.go, and follows to principle of
always returning a discrete type rather than an Interface. All callers
of "FakeMounter" are changed to instead use "NewFakeMounter()". The
FakeMounter "Log" struct member is changed to not be exported, and
instead only access through a new "GetLog()" method.
2019-11-08 08:07:41 -07:00
Alex Wang
b1a3fb4988 change kubelte nodeInfo to nodeLister 2019-10-24 22:46:59 +08:00
draveness
1163a1d51e feat: update taint nodes by condition to GA 2019-10-19 09:17:41 +08:00
Travis Rhoden
935c23f2ad
Move HostUtil to pkg/volume/util/hostutil
This patch moves the HostUtil functionality from the util/mount package
to the volume/util/hostutil package.

All `*NewHostUtil*` calls are changed to return concrete types instead
of interfaces.

All callers are changed to use the `*NewHostUtil*` methods instead of
directly instantiating the concrete types.
2019-08-30 10:14:42 -06:00
Jan Safranek
5b69362ff0 Skip unused volumes in VolumeManager
DesiredStateOfWorldPopulator should skip a volume that is not used in any
pod. "Used" means either mounted (via volumeMounts) or used as raw block
device (via volumeDevices).

Especially when block feature is disabled, a block volume must not get into
DesiredStateOfWorld, because it would be formatted and mounted there.
2019-08-15 10:48:38 +02:00
Travis Rhoden
be7da5052f Refactor pkg/util/mount to be more reusable
This patch refactors pkg/util/mount to be more usable outside of
Kubernetes. This is done by refactoring mount.Interface to only contain
methods that are not K8s specific. Methods that are not relevant to
basic mount activities but still have OS-specific implementations are
now found in a mount.HostUtils interface.
2019-06-14 09:35:18 -06:00
Tara Gu
5e18554442 Implement plugin manager - a controller that manages plugin registration/unregistration 2019-05-30 19:00:59 -04:00
Kubernetes Prow Robot
38468e4338
Merge pull request #70536 from mysunshine92/golint-hostpath
fix golint for pkg/volume/host_path
2019-05-29 11:25:46 -07:00
chaowang
9ba0d47465
Remove deprecated Kubelet security controls
Change-Id: I9f770e6780b60e881092690f6f177d726f474fe3
2019-05-16 13:33:52 -04:00
Andrew Kim
c919139245 update import of generic featuregate code from k8s.io/apiserver/pkg/util/feature -> k8s.io/component-base/featuregate 2019-05-08 10:01:50 -04:00
yameiwang
f130e48deb fix golint errors in pkg/volume/host_path 2019-04-25 06:46:47 +08:00
Davanum Srinivas
7b8c9acc09
remove unused code
Change-Id: If821920ec8872e326b7d85437ad8d2620807799d
2019-04-19 08:36:31 -04:00
Wei Huang
d67e7fd47f
kubelet: updated logic of verifying a static critical pod
- check if a pod is static by its static pod info
- meanwhile, check if a pod is critical by its corresponding mirror pod info
2019-03-12 23:40:20 -07:00
Travis Rhoden
2c4d748bed Refactor subpath out of pkg/util/mount
This patch moves subpath functionality out of pkg/util/mount and into a
new package pkg/volume/util/subpath. NSEnter funtionality is maintained.
2019-02-26 19:59:53 -07:00
David Ashpole
70a7fdda02 use Pod.Status.StartTime as pod's cgroup start time in summary API 2018-12-14 14:26:55 -08:00
yuexiao-wang
f3353c358d [scheduler cleanup phase 2]: Rename to
Signed-off-by: yuexiao-wang <wang.yuexiao@zte.com.cn>
2018-12-11 11:21:12 +08:00
Jordan Liggitt
4dca07ef7e Fixup incorrect use of DefaultFeatureGate.Set in tests 2018-11-21 11:51:33 -05:00
k8s-ci-robot
e85cb406eb
Merge pull request #65567 from ceshihao/pod_status_after_eviction
Pod status should contain ContainerStatuses if deadline exceeded
2018-10-17 11:56:41 -07:00
Christoph Blecker
0d63b55944
fix vet error in pkg/kubelet/kubelet_test.go
pkg/kubelet/kubelet_test.go:1573: *k8s.io/kubernetes/pkg/kubelet/container/testing.FakePod composite literal uses unkeyed fields
2018-10-05 15:52:44 -07:00
Cheng Pan
7d68fd00e6 fix golint for pkg/volume/gce_pd 2018-10-01 18:58:30 +00:00
k8s-ci-robot
a8e8e891f2
Merge pull request #68289 from denkensk/fix-simple-pkg-kubelet
Simple code fixed in in pkg/kubelet
2018-09-25 06:14:03 -07:00
Cheng Pan
000e30086b fix golint for pkg/volume/aws_ebs 2018-09-22 05:56:05 +00:00
wangqingcan
b0c308f082 Simple code and typo fixed in in pkg/kubelet 2018-09-05 21:51:32 +08:00
Pengfei Ni
376b45cb64 Fix unit tests for Windows
* TestMakeBlockVolume is moved to Linux only.
* TestMakeMounts are running on both Linux and Windows
2018-09-05 10:22:53 +08:00
Michael Taufen
1b7d06e025 Kubelet creates and manages node leases
This extends the Kubelet to create and periodically update leases in a
new kube-node-lease namespace. Based on [KEP-0009](https://github.com/kubernetes/community/blob/master/keps/sig-node/0009-node-heartbeat.md),
these leases can be used as a node health signal, and will allow us to
reduce the load caused by over-frequent node status reporting.

- add NodeLease feature gate
- add kube-node-lease system namespace for node leases
- add Kubelet option for lease duration
- add Kubelet-internal lease controller to create and update lease
- add e2e test for NodeLease feature
- modify node authorizer and node restriction admission controller
to allow Kubelets access to corresponding leases
2018-08-26 16:03:36 -07:00
Da K. Ma
aac9f1cbaa Taint node when initializing node.
Signed-off-by: Da K. Ma <klaus1982.cn@gmail.com>
2018-07-23 12:52:05 +08:00
Michael Taufen
f057c9a4ae port setNodeMemoryPressureCondition to Setter abstraction, add test 2018-07-16 09:09:47 -07:00
ceshihao
3b9ed9afff pod status should contain ContainerStatuses after eviction 2018-06-28 11:52:08 +08:00
Tim Allclair
5955b839ff
Cleanup verbose cAdvisor mocking in Kubelet unit tests 2018-06-27 11:53:41 -07:00
Kubernetes Submit Queue
1ca851baec
Merge pull request #64860 from wgliang/master.kubelet-check-limit
Automatic merge from submit-queue (batch tested with PRs 65290, 65326, 65289, 65334, 64860). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

checkLimitsForResolvConf for the  pod create and update events instead of checking period

**What this PR does / why we need it**:

- Check for the same at pod create and update events instead of checking continuously for every 30 seconds.
- Increase the logging level to 4 or higher since the event is not catastrophic to cluster health .


**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #64849

**Special notes for your reviewer**:
@ravisantoshgudimetla 

**Release note**:

```release-note
checkLimitsForResolvConf for the  pod create and update events instead of checking period
```
2018-06-22 04:43:16 -07:00
Kubernetes Submit Queue
aa25539ef6
Merge pull request #64451 from wgliang/master.remove-kubelet
Automatic merge from submit-queue (batch tested with PRs 64688, 64451, 64504, 64506, 56358). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

cleanup some dead kubelet code

**Release note**:

```release-note
NONE
```
2018-06-20 05:48:11 -07:00
Guoliang Wang
4f9d2047dd checkLimitsForResolvConf for the pod create and update events instead of checking period 2018-06-07 10:14:22 +08:00
Mike Danese
90ba15ee74 auth: standalone kubelets shouldn't start a token manager 2018-06-05 17:31:26 -07:00
Mike Danese
91feb345aa implement service account token projection 2018-06-04 17:22:08 -07:00
Hemant Kumar
1f9404dfc0 Implement kubelet side changes for writing volume limit to node
Add tests for checking node limits
2018-06-01 19:17:30 -04:00
Guoliang Wang
761cf41427 Move pkg/scheduler/schedulercache -> pkg/scheduler/cache 2018-05-31 22:55:34 +08:00
Guoliang Wang
9449a4372e cleanup some dead kubelet code 2018-05-29 22:38:01 +08:00
vikaschoudhary16
cedbd93255 Make 'pod' package to use unified checkpointManager
Signed-off-by: vikaschoudhary16 <choudharyvikas16@gmail.com>
2018-04-16 01:30:20 -04:00
Yu-Ju Hong
37d30a0815 Remove outdated network plugin code
The code was added to support rktnetes and non-CRI docker integrations.
These legacy integrations have already been removed from the codebase.
This change removes the compatibility code existing soley for the
legacy integrations.
2018-04-11 09:21:17 -07:00
Rohit Agarwal
87dda3375b Delete in-tree support for NVIDIA GPUs.
This removes the alpha Accelerators feature gate which was deprecated in 1.10.
The alternative feature DevicePlugins went beta in 1.10.
2018-04-02 20:17:01 -07:00
Jing Xu
b2e744c620 Promote LocalStorageCapacityIsolation feature to beta
The LocalStorageCapacityIsolation feature added a new resource type
ResourceEphemeralStorage "ephemeral-storage" so that this resource can
be allocated, limited, and consumed as the same way as CPU/memory. All
the features related to resource management (resource request/limit, quota, limitrange) are avaiable for local ephemeral storage.

This local ephemeral storage represents the storage for root file system, which will be consumed by containers' writtable layer and logs. Some volumes such as emptyDir might also consume this storage.
2018-03-02 15:10:08 -08:00
Yang Guo
8d880506fe Support cluster-level extended resources in kubelet and kube-scheduler
Co-authored-by: Yang Guo <ygg@google.com>
Co-authored-by: Chun Chen <chenchun.feed@gmail.com>
2018-02-27 17:25:30 -08:00
wackxu
f737ad62ed update import 2018-02-27 20:23:35 +08:00
Lantao Liu
d7b21a3358 Use container log manager in kubelet 2018-02-23 01:42:35 +00:00
Kubernetes Submit Queue
fb340a4695
Merge pull request #57824 from thockin/gcr-vanity
Automatic merge from submit-queue (batch tested with PRs 57824, 58806, 59410, 59280). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

2nd try at using a vanity GCR name

The 2nd commit here is the changes relative to the reverted PR.  Please focus review attention on that.

This is the 2nd attempt.  The previous try (#57573) was reverted while we
figured out the regional mirrors (oops).
    
New plan: k8s.gcr.io is a read-only facade that auto-detects your source
region (us, eu, or asia for now) and pulls from the closest.  To publish
an image, push k8s-staging.gcr.io and it will be synced to the regionals
automatically (similar to today).  For now the staging is an alias to
gcr.io/google_containers (the legacy URL).
    
When we move off of google-owned projects (working on it), then we just
do a one-time sync, and change the google-internal config, and nobody
outside should notice.
    
We can, in parallel, change the auto-sync into a manual sync - send a PR
to "promote" something from staging, and a bot activates it.  Nice and
visible, easy to keep track of.

xref https://github.com/kubernetes/release/issues/281

TL;DR:
  *  The new `staging-k8s.gcr.io` is where we push images.  It is literally an alias to `gcr.io/google_containers` (the existing repo) and is hosted in the US.
  * The contents of `staging-k8s.gcr.io` are automatically synced to `{asia,eu,us)-k8s.gcr.io`.
  * The new `k8s.gcr.io` will be a read-only alias to whichever regional repo is closest to you.
  * In the future, images will be promoted from `staging` to regional "prod" more explicitly and auditably.

 ```release-note
Use "k8s.gcr.io" for pulling container images rather than "gcr.io/google_containers".  Images are already synced, so this should not impact anyone materially.
    
Documentation and tools should all convert to the new name. Users should take note of this in case they see this new name in the system.
```
2018-02-08 03:29:32 -08:00
Tim Hockin
3586986416 Switch to k8s.gcr.io vanity domain
This is the 2nd attempt.  The previous was reverted while we figured out
the regional mirrors (oops).

New plan: k8s.gcr.io is a read-only facade that auto-detects your source
region (us, eu, or asia for now) and pulls from the closest.  To publish
an image, push k8s-staging.gcr.io and it will be synced to the regionals
automatically (similar to today).  For now the staging is an alias to
gcr.io/google_containers (the legacy URL).

When we move off of google-owned projects (working on it), then we just
do a one-time sync, and change the google-internal config, and nobody
outside should notice.

We can, in parallel, change the auto-sync into a manual sync - send a PR
to "promote" something from staging, and a bot activates it.  Nice and
visible, easy to keep track of.
2018-02-07 21:14:19 -08:00
Joel Smith
749980b726 Handle fetch of container logs of error containers during pod termination
* improve error returned when failing to fetch container logs
* handle cases where logs are requested for containers without the container ID
2018-02-07 12:23:56 -07:00
Kubernetes Submit Queue
5d9f1f82a6
Merge pull request #58229 from sjenning/exempt-sandbox-image
Automatic merge from submit-queue (batch tested with PRs 58422, 58229, 58421, 58435, 58475). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

kubelet: imagegc: exempt sandbox image

The image GC logic currently does not consider the sandbox image to be in-use by pods, since it isn't explicitly listed in the pod spec.  However, it is trivially in-use if there are any pods running on the node.

This change adds logic to exempt the sandbox image from GC by always considering it as in-use.

**Reviewer Note**
I am changing `(m *kubeGenericRuntimeManager) GetImageRef` to return the ID always rather than the first tag if it exists.  Seemed ok to me.  Makes some error messages a little less readable in that the ID will be printed and not the tag.  Just wanted to see what reviewers think about this.

@derekwaynecarr @dashpole
2018-01-18 16:36:36 -08:00
Seth Jennings
19a546758c kubelet: imagegc: exempt sandbox image 2018-01-17 15:10:44 -06:00
linweibin
fa8afc1d39 Remove unused code in UT files in pkg/ 2018-01-15 16:02:35 +08:00
Jonathan Basseri
30b89d830b Move scheduler code out of plugin directory.
This moves plugin/pkg/scheduler to pkg/scheduler and
plugin/cmd/kube-scheduler to cmd/kube-scheduler.

Bulk of the work was done with gomvpkg, except for kube-scheduler main
package.
2018-01-05 15:05:01 -08:00
Tim Hockin
e9dd8a68f6 Revert k8s.gcr.io vanity domain
This reverts commit eba5b6092a.

Fixes https://github.com/kubernetes/kubernetes/issues/57526
2017-12-22 14:36:16 -08:00
Tim Hockin
eba5b6092a Use k8s.gcr.io vanity domain for container images 2017-12-18 09:18:34 -08:00
Zihong Zheng
5915b87f8a Rearrange kubelet networking codes 2017-11-09 13:43:30 -08:00
Shawn Hsiao
5cba1f47c3 Support copying options in resolv.conf into pod sandbox when dnsPolicy is Default 2017-11-07 07:54:52 -05:00
Zihong Zheng
c1a959c660 Clean up redundant DNS related kubelet codes
Signed-off-by: Zihong Zheng <zihongz@google.com>
2017-11-04 12:33:32 -07:00
lichuqiang
0630896383 update unit test for plugin resources allocation reinforcement 2017-11-02 09:18:24 +08:00