· 7 years ago · Mar 23, 2018, 02:28 PM
1kqueen package
2==============
3
4.. toctree::
5 :maxdepth: 3
6 :caption: Contents:
7
8 kqueen.engines
9
10Models
11-------
12
13.. automodule:: kqueen.models
14 :members:
15 :undoc-members:
16 :show-inheritance:
17
18
19Kubernetes API
20----------------------
21
22.. automodule:: kqueen.kubeapi
23 :members:
24 :undoc-members:
25 :show-inheritance:
26
27Server
28---------------------
29
30.. automodule:: kqueen.server
31 :members:
32 :undoc-members:
33 :show-inheritance:
34
35Helpers
36-----------------------
37
38.. automodule:: kqueen.helpers
39 :members:
40 :undoc-members:
41 :show-inheritance:
42
43Serializers
44--------------------------
45
46.. automodule:: kqueen.serializers
47 :members:
48 :undoc-members:
49 :show-inheritance:
50
51
52Configuration
53--------------
54
55Sample configuration files are located in the `config/` directory. The default configuration file is `config/dev.py`.
56To define a different configuration file, set the `KQUEEN_CONFIG_FILE` environment variable. To override the values defined
57in the configuration file, set the environment variable matching the KQUEEN_<config_parameter_name> pattern.
58
59.. list-table:: Configuration options
60 :header-rows: 1
61
62
63 * - Name
64 - Default
65 - Description
66
67 * - CONFIG_FILE
68 - config/dev.py
69 - Configuration file to load during startup
70
71 * - DEBUG
72 - False
73 - Debug mode for flask and all loggers
74
75 * - SECRET_KEY
76 - None
77 - This key is used for server-side encryption (cookies, secret database fields) and must be at least 16 characters in length.
78 * - ETCD_HOST
79 - localhost
80 - Hostname address of the etcd server
81 * - ETCD_PORT
82 - 4001
83 - Port for etcd server
84 * - ETCD_PREFIX
85 - /kqueen
86 - Prefix URL for objects in etcd
87
88
89 * - JWT_DEFAULT_REALM
90 - Login Required
91 - The default realm
92 * - JWT_AUTH_URL_RULE
93 - /api/v1/auth
94 - Authentication endpoint returning token.
95 * - JWT_EXPIRATION_DELTA
96 - timedelta(hours=1)
97 - JWT token lifetime.
98
99 * - JENKINS_ANCHOR_PARAMETER
100 - STACK_NAME
101 - This parameter is used to match Jenkins builds with clusters.
102 * - JENKINS_API_URL
103 - None
104 - REST API for Jenkins
105 * - JENKINS_PASSWORD
106 - None
107 - Optional. The default Jenkins password. It can be overridden by another value specified in the request.
108 * - JENKINS_PROVISION_JOB_CTX
109 - {}
110 - Dictionary for predefined Jenkins job context
111 * - JENKINS_PROVISION_JOB_NAME
112 - deploy-aws-k8s_ha_calico_sm
113 - Name of the Jenkins job used to deploy a cluster.
114 * - JENKINS_USERNAME
115 - None
116 - Optional. The default Jenkins username. It can be overridden by another value specified in the request.
117
118 * - CLUSTER_ERROR_STATE
119 - Error
120 - Caption for a cluster in error state.
121 * - CLUSTER_OK_STATE
122 - OK
123 - Caption for a cluster in OK state.
124 * - CLUSTER_PROVISIONING_STATE
125 - Deploying
126 - Caption for a cluster in provisioning state.
127 * - CLUSTER_DEPROVISIONING_STATE
128 - Destroying
129 - Caption for a cluster in deprovisioning (deleting) state.
130 * - CLUSTER_UNKNOWN_STATE
131 - Unknown
132 - Caption for a cluster with unknown state.
133
134 * - CLUSTER_STATE_ON_LIST
135 - True
136 - Update the state of clusters on cluster list. This can be disabled for organizations with a large number of clusters in the deploy state.
137
138 * - PROVISIONER_ERROR_STATE
139 - Error
140 - Caption for errored provisioner.
141 * - PROVISIONER_OK_STATE
142 - OK
143 - Caption for working provisioner.
144 * - PROVISIONER_UNKNOWN_STATE
145 - Not Reachable
146 - Caption for unknown provisioner.
147 * - PROVISIONER_ENGINE_WHITELIST
148 - None
149 - Enable only engines in list.
150
151 * - PROMETHEUS_WHITELIST
152 - 127.0.0.0/8
153 - Addresses allowed to access metrics endpoint without token
154
155Before you provision a Kubernetes cluster, you may need to deploy and configure the following external services:
156
157External services deployment
158-------
159* **Members managing** - To provide user registration by invitations from other users by emails, mail server should be set up.
160It is possible to use kqueen predefined mail service or to run new one.
161
162* **Metrics collecting** - Its possible to add external Prometheus serveri to extend monitoring in Kqueen. In case of using external server, need to include rules from kqueen/prod/prometheus into existing Prometheus service. Otherwise, its possible to use predefined Prometheus, defined in docker-compose.production.yml.
163Configure prometheus service (ip address/port, volumes) in docker-compose.production.yml. Define scraper IP address for PROMETHEUS_WHITELIST variable in kqueen api service section.
164All application metrics exported on **/metrics** api endpoint and this metric can scraped by any external Prometheus instance.
165
166**To set up members management:**
167
1681. Define mail service into docker compose file.
169
170Example configuration mail service in docker-compose.production.yml:
171.. code-block:: yaml
172 mail:
173 image: modularitycontainers/postfix
174 volumes:
175 - /var/spool/postfix:/var/spool/postfix
176 - /var/mail:/var/spool/mail
177 environment:
178 MYHOSTNAME: 'mail'
179
1802. Configure following variables in kqueen ui service section:
181
182.. code-block:: yaml
183 KQUEENUI_MAIL_SERVER: mail
184 KQUEENUI_MAIL_PORT: 10025
185
186Pay attention that volume-mapping for mail containers is additional feature. It used for store mailing history and forward additional postfix mail configuration (it should configured properly on local machine),
187otherwise its possible to run mail server without volume-mapping. After all configuration steps, user should invite members using email notification. Member should receive mail with activation link to Kqueen service and possibility of password-setting.
188Superadmin also can manage member roles.
189
190**To set up metrics collecting:**
1911. Configure prometheus service (ip address/port, volumes) in docker-compose.production.yml.
192
193Example configuration mail service in docker-compose.production.yml:
194.. code-block:: yaml
195 prometheus:
196 image: prom/prometheus
197 restart: always
198 ports:
199 - 127.0.0.1:9090:9090
200 volumes:
201 - ./prod/prometheus/:/etc/prometheus/:Z
202 - /mnt/storage/kqueen/prometheus/:/prometheus/
203 links:
204 - api
205 - etcd
206
2072. Define Prometheus scraper IP address in kqueen api service section:
208
209.. code-block:: yaml
210 KQUEEN_PROMETHEUS_WHITELIST: '172.16.238.0/24'
211
212There two ways to get metrics:
213
214- Kqueen API
215.. code-block:: bash
216 TOKEN=$(curl -s -H "Content-Type: application/json" --data '{"username":"admin","password":"default"}' -X POST <<kqueen_api_host>>:5000/api/v1/auth | jq -r '.access_token'); echo $TOKEN
217 curl -H "Authorization: Bearer $TOKEN" <<kqueen_api_host>>:5000/metrics/
218
219- Prometheus API
220Add scraper IP address to PROMETHEUS_WHITELIST configuration and then:
221
222.. code-block:: bash
223 curl <<prometheus_host>>:<<prometheus_port>>/metrics
224
225
226Provisioner configuration settings
227-------
228You can provision a Kubernetes cluster using various community of engines, such as:
229
230* Provision a Kubernetes cluster using the Google Kubernetes Engine:
231
232**Google Kubernetes Engine**
2331. Register account on GCE (https://console.cloud.google.com)
2342. Create your project
2353. Go to ```API’s & Services``` tab
2364. Go ```Credentials``` tab
2375. Click ```Create credentials```
2386. Choose ```Service Account key```
2397. Choose your Service account
2408. Choose Json format for key
2419. Download it
24210. On Kqueen UI choose ```Create Provisioner```
24311. Choose ```Google Kubernetes Engine```
24412. Insert your project ID (```Project info``` tab on the main page of GCE Dashboard https://console.cloud.google.com )
24513. Insert downloaded Json snippet, which contains key and submit provisioner creation
24614. On Kqueen UI choose ```Deploy Cluster```
24715. Choose defined GCE provisioner
24816. Specify cluster requirements
24917. Click ```Submit```
25018. Watch for cluster status from Kqueen main dashboard
251
252* Provision a Kubernetes cluster using the Azure Kubernetes Service:
253
254**Azure Kubernetes Service**
2551. Login into https://portal.azure.com with your azure account
2562. Follow official Microsoft guide https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-create-service-principal-portal#create-an-azure-active-directory-application
2573. Get Application ID, Application Secret, Tenant ID (Directory ID), Subscription ID
2584. Pay attention to set ‘Owner’ role to your Application in Subscription settings to provide possibility of creating k8s clusters. (see latest steps in p.2). Save Application secret manually, because it will be unavailable from Azure Web UI after generating.
2595. Create Resource group from ‘Resource groups’ tab and get ‘Resource group name’
2606. Check that Application had ‘Owner’ role in Resource group. (go to ‘Resource groups’ -> your_group -> Access Control(IAM))
2617. Go to Kqueen ‘Create provisioner’ tab and choose AKS engine
2628. Set ‘Client ID’ as Application ID from p.3
2639. Set ‘Resource group name’ as ‘Resource group name’ from p.4
26410 Set ‘Secret’ as Application Secret from p.3
26511. Set ‘Subscription ID’ as Subscription ID from p.3
26612. Set ‘Tenant ID’ as Tenant(Directory) ID from p.3
26713. On Kqueen UI choose ```Deploy Cluster```
26814. Choose defined AKS provisioner
26915. Specify cluster requirements
27016. Specify public ssh key for connect to AKS vm’s *
27117. Click ```Submit```
27218. Watch for cluster status from Kqueen main dashboard
273
274**Pay attention** that Admin Console in Azure portal supported properly only in IE and Microsoft Edge, and may fail in Safari, Opera, Chrome, Firefox due MIcrosoft issues, like
275https://microsoftintune.uservoice.com/forums/291681-ideas/suggestions/18602776-admin-console-support-on-mac-osx
276
277**Pay attention** that AKS created separate resource during cluster creating (it used defined RG as prefix ). It may affect your billing. For ex.:
278Your group : Kqueen
279Cluster-generated RG: MC_Kqueen_44a37a65-1dff-4ef8-97ca-87fa3b8aee62_eastus
280Referenced to https://github.com/Azure/AKS/issues/3
281Docs: https://docs.microsoft.com/en-us/azure/aks/faq#why-are-two-resource-groups-created-with-aks
282
283**Pay attention** for connect to AKS vm’s, need to:
284- assign public ip address to vm (example: https://gist.github.com/naumvd95/576d6e48200597ca89b26de15e8d3675)
285- define valid public ssh key in p.8
286- ssh azureuser@<<public_ip>> -i .ssh/your_defined_id_rsa
287
288
289* **Manual engine for attach existing clusters**
290 Define valid kubernetes configuration file in ```Create Cluster``` tab to attach existing Kubernetes cluster in read-only mode.
291
292Backup
293-------
294
295The only one statefull component of Kqueen is the etcd and users should follow https://coreos.com/etcd/docs/latest/v2/admin_guide.html#disaster-recovery. We are using `v2` etcd keys so example backup workflow can be:
296
297::
298
299 # Backup etcd to directory /root/backup/ (etcd data stored in /var/lib/etcd/default)
300 etcdctl backup --data-dir /var/lib/etcd/default --backup-dir /root/backup/
301
302Recovery
303
304::
305
306 # Move data to etcd directory
307 mv -v /root/backup/* /var/lib/etcd/default/
308
309 # Start new etcd with these two extra parameters (among the other)
310 # for example: etcd --force-new-cluster
311
312
313kqueen