n7hdWZVS

· 8 years ago · Apr 06, 2018, 01:28 PM
1
2Skip to end of metadata
3Created by Sebastian Mincewicz, last modified on Apr 07, 2017 Go to start of metadata
4Infrastructure architecture
5S3 bucket structure
6Scheduled events
7AWS accounts and environments
8Security
9Logging
10CloudWatch logs
11Metrics
12Management environment
13GIT repositories
14Terraform
15Known bugs and unsupported AWS features
16TF Scaffold script
17TFState files
18Manifest file
19'motr-terraform' module
20Structure
21Switches
22Jenkins jobs
23MOTR_APP_build
24MOTR_INT_build
25MOTR_APP_promote
26MOTR_APP_terraform_run
27MOTR_APP_loaddata
28MOTR_APP_Loader_perftest
29MOTR_APP_Notifier_perftest
30MOTR_DEV_lull
31MOTR_MGMT_terraform_run
32MOTR_NFT_JMeter
33MOTR_security_burp_tests
34HOW-TO's
35Create a new live environment in the AWS Live account
36Destroy a live environment
37Rollback PRD environment to previously deployed artefacts
38Encrypt/decrypt a key with KMS
39DynamoDB backup/restore
40Additional information
41Error page
42CloudFront vs. S3 Bucket origin
43 
44
45 
46
47Infrastructure architecture
48MOTR infrastructure is fully serverless which means that there is no need to manage any underlying infrastructure like ec2 instances and there is no VPC related (networking) configuration.
49
50All  services used to build the MOTR are located in the AWS public cloud but access to them is either limited on the AWS account level or with use of appropriate IAM roles/policies.
51
52MOTR infrastructure consist of two parts:
53
541/ FRONTEND (Online) - allows visitors to subscribe and unsubscribe for reminders, including confirmation links.
55
562/ BACKEND (Offline) - does the subscription processing work, daily determines which subscriptions should get a reminder and delivers reminders via GOV.UK Notify.
57
58MOTR is integrated with two 3rd party services:
59
601/ GOV.UK Notify to notify subscribers using email and potentially text messages in the future.
61
622/ MOTH Trade API to retrieve information about vehicle tests and their expiry dates.
63
64
65For details regarding what the data flows are and how particular services work please refer to MOTR General documentation https://wiki.i-env.net/display/MOTR.
66
67S3 bucket structure
68The name used for MOTR S3 buckets is a combination of 'uk.gov.dvsa.motr.' and environment name, ex. uk.gov.dvsa.motr.demo or uk.gov.dvsa.motr.nft.
69
70This is how the S3 bucket folder structure looks like:
71
72assets
73images
74javascripts
75stylesheets
76assets_archive (zipped assets, for eventual rollback)
77lambdas (zipped lambda functions, deployed with terraform)
78logs (only for environments for which logs do not have to persist)
79terraform
80manifest_<manifest version>.tfvars (manifest files, for details see below Terraform > Manifest file)
81<artefact name>_last_commit (only in DEV environments, used for checking if assets/lambdas have changed and have to be rebuilt)
82Scheduled events
83WebApp warmup - a Lambda based mechanism (NPinger) that keeps a number of Lambda containers warm to prevent cold starts. 
84
85Subscription processing triggers - CloudWatch events with defined schedules are used to trigger backend processing of MOTR.
86
87
88
89 
90
91AWS accounts and environments
92MOTR uses two separate AWS accounts, one for live environments and another one for development environments. The development account is the same account used for MOT development environments.
93
94MOTR Live account - https://153412556428.signin.aws.amazon.com/console (https://dvsamotrprd.signin.aws.amazon.com/console)
95
96Environments:
97
98PP (Pre-production)
99PRD (Production)
100MOT(R) Dev account - https://054631451206.signin.aws.amazon.com/console (https://dvsamotdev.signin.aws.amazon.com/console)
101
102Environments:
103
104INT (Integration)
105DEMO
106NFT
107A bunch of other environments named with the first names of developers
108In this document, we refer to complete, non-overlapping deployments of MOTR resources as environments. Because of the way our deployment code can manipulate environment parameters, several environments can co-exist in full separation under the same AWS account with no impact on each other. Terraform code used for building MOTR environments gives the ability to have either a full-stack (with CloudFront) infrastructure or dev-stack infrastructure (without CloudFront). This duality was introduced because of the time needed to deploy a CloudFront distribution (around 28 minutes) that would heavily degrade dev teams performance during the development phase and wouldn't allow putting dev environments to sleep every evening in an easy way.
109
110Full-stack environments (with CloudFront)
111
112PRD
113PP
114NFT
115DEMO
116Dev-stack environments (without CloudFront)
117
118INT
119Developers' environments
120
121Dev-stack environments are not integrated with any real MOTH Trade API endpoint and use MOTH Mock instead (a static json response configured on the API Gateway integration response level).
122The entry point for DEV environments is the API Gateway URL and for LIVE environments it is the Route53 alias:
123
124PRD: https://www.reminders.mot-testing.service.gov.uk
125PP: https://pp.reminders.mot-testing.service.gov.uk
126DEMO: https://demo.dev.reminders.mot-testing.service.gov.uk
127NFT: https://nft.dev.reminders.mot-testing.service.gov.uk
128INT: https://o6zid9xpei.execute-api.eu-west-1.amazonaws.com/int/ (where 'o6zid9xpei' ID part is being generated with every (re)creation of the API Gateway deployment)
129 
130
131Security
132Security mechanisms used across the MOTR infrastructure that define how services can communicate between each other:
133
134CloudFront â†’ API key â†’ API Gateway
135CloudFront â†’ Origin Access Identity / Bucket policy â†’ S3 bucket
136API Gateway â†’ IAM role â†’ Lambda
137Lambda â†’ IAM role â†’ DynamoDB
138Lambda â†’ IAM role â†’ SQS
139Lambda â†’ IAM role â†’ KMS
140Lambda â†’ IAM role â†’ CloudWatch
141Lambda â†’ API key â†’ MOTH Trade API (3rd party service)
142Lambda â†’ Security token â†’ GOV.UK Notify (3rd party service)
143Additionally, Api key and security token used for communication with external services are encrypted with KMS so their values are not exposed in the source code.
144
145
146
147Additional CloudFront security features available:
148
149WAF with whitelisted IPs (used currently only for DEMO, NFT, PP)
150GEO-Location (disabled)
151
152Even though the API Gateway cannot be integrated with WAF currently, DEV environments do not need to be secured by other means as they are short living environments that are not integrated with any real 3rd party service + they get destroyed automatically every evening.
153
154Logging
155CloudWatch logs
156The only CloudWatch logs being generated are Lambda logs (json formatted) that are grouped per environment and per Lambda.
157
158Terraform implementation allows you to decide if those CloudWatch logs should be maintained or not (if they should still persist after an environment gets destroyed) and how they should expire.
159
160CloudWatch logs are not being streamed to any external logs processing service and can be queried only using AWS console or AWC CLI.
161
162The log level can be changed by modifying the Lambda LOG_LEVEL environment variable value with terraform.
163
164MotrWebHandler
165
166
167
168MotrSubscriptionLoader
169
170
171
172MotrNotifier
173
174
175
176 
177
178 CloudFront access logs
179
180CloudFront can create log files that contain detailed information about every user request that CloudFront receives. By enabling logging a S3 bucket has to be specified as a destination where logs can be stored. It is not recommended to use the same bucket that is used as an origin.
181
182For PRD and optionally for PP environments there is a 'uk.gov.dvsa.motr.logs' bucket created in the MGMT environment where logs can be streamed to. 'logging_bucket' variable should be used to specify the bucket (its value is empty by default what will cause that the bucket used as the origin will be used).
183
184$ aws s3 ls s3://uk.gov.dvsa.motr.nft/logs/cloudfront/
1852017-03-21 17:23:26       1266 E3M7P0NJ1MGC2J.2017-03-21-17.53e6f38b.gz
1862017-03-21 17:32:51       2262 E3M7P0NJ1MGC2J.2017-03-21-17.73946a7f.gz
1872017-03-22 10:21:50       2489 E3M7P0NJ1MGC2J.2017-03-22-10.52ffaf26.gz
1882017-03-22 10:11:49       1270 E3M7P0NJ1MGC2J.2017-03-22-10.5ec5e8c3.gz
189More information about the CloudFront access logs can be found here: http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/AccessLogs.html.
190
191 
192
193Metrics
194For metrics related information please refer to CloudWatch metrics.
195
196 
197
198Management environment
199AWS services located in the MGMT environment:
200
201Route53 hosted zone for 'reminders.mot-testing.service.gov.uk'
202KMS keys per MOTR environment
203S3 logging bucket for CloudFront access logs
204WAF configuration for some of the MOTR LIVE environments
205ACM (SES/SNS) for issuing certificates for MOTR LIVE environments
206
207Keeping KMS bits in the MGMT environment rather then per MOTR APP environment does not require encrypted keys/tokens to be encrypted again with different keys whenever an environment gets destroyed.
208
209ACM for CloudFront is supported only in the N. Virginia AWS region (us-east-1).
210
211
212
213Development environments use MOT development AWS account so management services used by them are defined in the MOT Terraform code (FB MGMT environment).
214
215 
216
217GIT repositories
218MOTR code is meant to be public so all the repositories are being hosted on GitHub. The only Git repository maintained on the internal GitLab server is the webops/motr repo that contains MOTR environmental configurations, scripts and jenkinsfiles used by Jenkins for automation mechanisms allowing us to build and tests MOTR environments.
219
220GitHub repos:
221
222https://github.com/dvsa/motr-terraform - MOTR Terraform module used to build an environment
223https://github.com/dvsa/motr-webapp - Java code for MOTR WebApp Lambda
224https://github.com/dvsa/motr-web-static - MOTR Web Static assets stored in S3 bucket and referenced by MOTR WebApp Lambda
225https://github.com/dvsa/motr-subscription-loader - Java code for MOTR Subscription Loader Lambda
226https://github.com/dvsa/motr-notifier - Java code for MOTR Notifier Lambda
227GitLab repos:
228
229https://gitlab.motdev.org.uk/webops/motr - MOTR environmental configurations + MOTR MGMT environment code + automation tools and Jenkins job configurations
230 
231
232Terraform
233Terraform is the only automation tool used for managing MOTR environments.
234
235Known bugs and unsupported AWS features
236Here is the list of various Terraform issues we had to or still have to fight and go around some of them in order to have things working.
237
238Some of these issues or lack of AWS features coverage require from us a manual intervention to be able to keep state consistency.
239
2401.	https://github.com/hashicorp/terraform/issues/6613	NO	aws_api_gateway_deployment resource doesn't get updated after changes to underlying requests/responses/methods/integrations	GLOBAL	Manual stage deployment
2412.	https://github.com/hashicorp/terraform/issues/9877	NO	change to aws_api_gateway_integration resource destroys all aws_api_gateway_integration_response resources	DEV environments only	No
2423.	https://github.com/hashicorp/terraform/issues/10335	NO	WAF - "The input token is no longer current."	MGMT environment	Rerun terraform
2434.	https://github.com/hashicorp/terraform/issues/10957	YES	aws_acm_certificate does not lookup certificates in different region	GLOBAL	Provide cert arn as input parameter
2445.	https://github.com/hashicorp/terraform/issues/11395	NO	aws_api_gateway_method_response resource concurrency clinch when using multiple response code	GLOBAL	Create dependencies structure in TF
2456.	https://github.com/hashicorp/terraform/issues/12233	NO	aws_api_gateway_integration resources not getting updated properly/permanently	GLOBAL	N/A (not needed)
2467.	https://github.com/hashicorp/terraform/issues/12297	NO	DynamoDB index non_key_attributes changes remain unrecognised	GLOBAL	Recreate DB table
2478.	https://github.com/hashicorp/terraform/issues/12593	NO	updating environment variables does not cause a new AWS Lambda version to be published	GLOBAL	Do not use this feature
248TF Scaffold script
249Standardisation aspects require that we use TF Scaffold script (supported by BJSS/TSS) as a wrapper/library for any terraform tool execution across all Jenkins jobs we use for MOTR.
250
251TF Scaffold script can be found here: https://github.com/cartest/terraformscaffold.
252
253TF Scaffold bootstrap script has been used to create a S3 bucket where all tfstate files per environments are stored. The -b/--bucket-prefix switch value used for MOTR is 'uk.gov.dvsa.motr'.
254
255By using TF Scaffold script it is not always possible to switch to the newest version of Terraform immediately because this very dynamically changing tool requires sometimes various adjustments to be made on the script side first.
256GitLab webops/motr repo structure:
257
258bin(temporary location of the TF Scaffold scripts)
259components
260mgmt (MOTR MGMT terraform code)
261motr_app (MOTR APP component calling MOTR module with a set of variables)
262etc
263env_<aws region>_<env>.tfvars (environment-specific variables)
264jenkinsfiles
265jenkinsfiles used in various Jenkins jobs (for details see below Jenkins jobs)
266perf
267various scripts used for performance testing
268s3
269terraform
270manifest.tfvars (MOTR manifest template used when creating a new environment only)
271scripts
272compare_file_variables.sh (scripts used to verify manifest key structure with the template file - used mostly during the development phase when keys were being added/changed)
273int_build_trigger.sh (scripts used by MOTR_INT_trigger job, for details see Jenkins jobs > MOTR_INT_trigger)
274TFState files
275TF Scaffold usage impose the way how Terraform tfstate files are being treated. The TF Scaffold bootstrap stage created a dedicated S3 bucket for storing tfstate files in fully separated folders.
276
277Both LIVE and DEV AWS accounts contain a bucket with name 'uk.gov.dvsa.motr-<account number>-eu-west-1' and where tfstate files are being downloaded, queried and updated with every build.
278
279Manifest file
280The manifest files is a concept that allows dynamic change of terraform variables without a need of manual changes being performed that would have to be merged to the webops/motr repo before running terraform.
281
282Sample content: 
283
284static_assets_hash = "85531f25f7ef854946a82ea128590bca0d0609bc"
285MotrWebHandler_s3_key = "motr-web-20170317125846-9439d343c43dc894c60a597e4bdae67fa776d4fe-15.zip"
286MotrSubscriptionLoader_s3_key = "motr-subscription-loader-20170316090521-7e31399b2ec64e9426cf910bb1c34399dceb0d43-15.zip"
287MotrNotifier_s3_key = "motr-notifier-20170307101225-8856559d18c1df261709b0b2f97abcf64c86305b-15.zip"
288Key/values stored in the manifest file define what versions of artefacts are currently used on a specific environment. 
289
290Every terraform command being executed during any of the jenkins jobs described below point to the manifest file, either latest or a specific version, in order to pass demanded values for artefacts keys.
291
292The promotion process relies on this manifest file and its version in order to deploy a particular version of artefacts set from one environment to another.
293
294The manifest file uses the following naming convention: manifest_<manifest version>.tfvars.
295
296The Manifest file version consist of the timestamp and the build number, ex. 20170320113478_1423.
297'motr-terraform' module
298The module used for building MOTR environments is public so by default it does not contain any static values. If any static value found it is because it was decided that it must by specified to ensure that everything works properly.
299
300Per module definition there is only one code no matter if you create a development environment or a live environment however there are couple of switches in the code based on witch some resources get created and some of them not.
301
302Structure
303The code is divided per resource group which means that every tf file represents a set of resources of the same AWS service. Additionally, there is an iam_policies folder that contains policies definitions used in the TF code.
304
305 
306
307.
308â”œâ”€â”€ apigateway.tf
309â”œâ”€â”€ cloudfront.tf
310â”œâ”€â”€ cloudwatch.tf
311â”œâ”€â”€ dynamodb.tf
312â”œâ”€â”€ iam.tf
313â”œâ”€â”€ iam_policies
314â”‚   â”œâ”€â”€ apig_assumerole_policy.json.tpl
315â”‚   â”œâ”€â”€ lambda_assumerole_policy.json.tpl
316â”‚   â”œâ”€â”€ lambda_notifier_permissions_policy.json.tpl
317â”‚   â”œâ”€â”€ lambda_subscr_loader_permissions_policy.json.tpl
318â”‚   â””â”€â”€ lambda_webapp_permissions_policy.json.tpl
319â”œâ”€â”€ lambda.tf
320â”œâ”€â”€ outputs.tf
321â”œâ”€â”€ providers.tf
322â”œâ”€â”€ route53.tf
323â”œâ”€â”€ s3.tf
324â”œâ”€â”€ sqs.tf
325â””â”€â”€ variables.tf
326
327Switches
328A list of switches (terraform boolean variables) based on which some resources get created and/or some options get enabled.
329
330with_cloudfront	
331CloudFront on/off switch.
332
333***It remains 'false' for all DEV environments!
334
335false	
336adds CloudFront distribution
337adds Route53 alias
338modifies value for STATIC_ASSETS_URL lambda environment variable
339adds API Key for API Gateway
340adds Usage plan for API Gateway
341sets api_key_required property value to 'true'
342modifies S3 bucket policy identifier
343public_dns_domain
344alias_record_name
345certificate_arn
346cf_apig_channel_key
347base_url
348waf_acl_id
349bucket_versioning_enabled	S3 bucket versioning switch.	false	N/A	N/A	N/A
350manage_cw_lg_web_lambda	Enable CloudWatch Log Group for MOTR WebHandler Lambda to be managed by Terraform.	false	If true - CloudWatch logs for WebApp lambda will be removed when an environment gets destroyed.	N/A	N/A
351manage_cw_lg_subscr_lambda	Enable CloudWatch Log Group for MOTR Subscription Loader Lambda to be managed by Terraform.	false	If true - CloudWatch logs for Subscription Loader lambda will be removed when an environment gets destroyed.	N/A	N/A
352manage_cw_lg_notifier_lambda	Enable CloudWatch Log Group for MOTR Notifier Lambda to be managed by Terraform.	false	If true - CloudWatch logs for Notifier will be removed when an environment gets destroyed.	N/A	N/A
353motr_loader_enabled	Whether the CloudWatch event rule should be enabled.	false	Subscription Loader lambda will be triggered periodically based on a schedule specified.	N/A	
354motr_loader_schedule
355motr_notifier_enabled	Whether the CloudWatch event rule should be enabled.	false	Notifier lambda will be triggered periodically based on a schedule specified.	N/A	
356motr_notifier_schedule
357MotrWebHandler_publish	WebApp Lambda publish switch.	false	When new WebApp lambda function will get deployed it will publish a new version.	N/A	
358MotrWebHandler_ver
359MotrSubscriptionLoader_publish	MotrSubscriptionLoader Lambda publish switch.	false	When new Subscription Loader lambda function will get deployed it will publish a new version.	N/A	
360MotrSubscriptionLoader_ver
361MotrNotifier_publish	MotrNotifier Lambda publish switch.	false	When new Notifier lambda function will get deployed it will publish a new version.	N/A	
362MotrNotifier_ver
363web_enable_warmup	Enable WebLambda for WarmUp.	false	Enables a CloudWatch event that will ping WebApp lambda periodically based on a schedule.	N/A	
364web_warmup_rate
365
366Jenkins jobs
367All MOTR related jenkins jobs can be found here: https://jenkins2.motdev.org.uk/view/MOTR.
368
369All Jenkins jobs specified below share the same set of reusable definitions: https://gitlab.motdev.org.uk/webops/motr/blob/master/jenkinsfiles/definitions.groovy.
370
371MOTR_APP_build
372This job is being used by developers to create and rebuild DEV environments.
373
374Average build time ~ 7 minutes.
375
376
377
378Git URL: https://gitlab.motdev.org.uk/webops/motr/blob/master/jenkinsfiles/app_build.groovy
379
380Job URL: https://jenkins2.motdev.org.uk/view/MOTR/job/MOTR_APP_build/
381
382Stages:
383
384checkout - all repositories that are used in the MOTR building process (see GIT repositories above) are being fetched
385verify env - it is being verified if the environment already exists and if yes then the manifest file is being obtained from the S3 bucket
386DB/SQS cleanup - DynamoDB tables are being deleted and SQS queue is being purged as a requirement for integration tests
387build assets - static assets (IF CHANGED) are being zipped and its version updated within the manifest file
388build lambdas - lambda functions for webapp, subscription-loader and notifier (IF CHANGED) are being rebuilt and their versions updated within the manifest file
389terraform plan - terraform execution plan is being created
390go? - a gate after the previous stage with timeout set to 5 minutes that gives the opportunity to verify if the execution plan reflects expected changes
391s3 create/verify - based on the output from stage 2, if the environment does not exist, a targeted terraform apply is being run in order to create the S3 bucket required in the next stage
392s3 synchronise - artefacts built and stored in the workspace are being synchronised with the S3 bucket as they have to be accessible before the next stage
393terraform apply - changes are being applied as per terraform plan and the manifest file is being uploaded to the S3 bucket
394tests - a bunch of tests is being executed:
395subscription-loader integration tests
396subscription-notifier integration tests
397selenium tests
398db-integration-webapp tests
399After the 'terraform apply' stage of every build a manifest file content is being printed out along with its version.
400
401#####################################################################################################
402MANIFEST VERSION: 20170320100028_15
403#####################################################################################################
404static_assets_hash = "85531f25f7ef854946a82ea128590bca0d0609bc"
405MotrWebHandler_s3_key = "motr-web-20170317125846-9439d343c43dc894c60a597e4bdae67fa776d4fe-15.zip"
406MotrSubscriptionLoader_s3_key = "motr-subscription-loader-20170316090521-7e31399b2ec64e9426cf910bb1c34399dceb0d43-15.zip"
407MotrNotifier_s3_key = "motr-notifier-20170307101225-8856559d18c1df261709b0b2f97abcf64c86305b-15.zip"
408#####################################################################################################
409Additionally, the manifest version is being displayed as a part of every build description. 
410
411
412
413MOTR_INT_build
414This job is being used to rebuild the INT environment automatically on every merge to master to any of the MOTR GitHub repositories.
415
416Because of the fact that GitHub is an external/public service that is not allowed to trigger anything in DVSA private AWS infrastructure, there is MOTR_INT_trigger job that queries those GitHub repositories for changes and if found it triggers the MOTR_INT_build job.
417Git URL: https://gitlab.motdev.org.uk/webops/motr/blob/master/jenkinsfiles/app_build.groovy
418
419Job URL: https://jenkins2.motdev.org.uk/view/MOTR/job/MOTR_INT_build/
420
421The job configuration and its stages are exactly the same as MOTR_APP_build, the only difference is that all branches for the INT environment are set to 'master' and they cannot be changed on fly.
422
423MOTR_APP_promote
424This job is being used to promote artefact from one environment to another.
425
426Average build time ~ 40 seconds.
427
428
429
430Git URL: https://gitlab.motdev.org.uk/webops/motr/blob/master/jenkinsfiles/app_promote.groovy
431
432Job URL: https://jenkins2.motdev.org.uk/view/MOTR/job/MOTR_APP_promote/
433
434For detailed information about the MOTR promotion process go MOT Reminders Application Promotion process
435
436MOTR_APP_terraform_run
437This job can be used to just run terraform plan-apply or plan-destroy against a particular MOTR environment.
438
439Average build time ~ 30 seconds.
440
441
442
443Git URL: https://gitlab.motdev.org.uk/webops/motr/blob/master/jenkinsfiles/app_terraform.groovy
444
445Job URL: https://jenkins2.motdev.org.uk/view/MOTR/job/MOTR_APP_terraform_run/
446
447This is a short version of the MOTR_APP_build job that does not contain any building or testing stages and also can be used to restore any previous versions of artefacts set (using manifest file version) deployed in the past.
448
449MOTR_APP_loaddata
450This is job is being used to populate DynamoDB with some data used for performances tests.
451
452Script used: XXXXX
453
454Git URL: XXXXX
455
456Job URL: https://jenkins2.motdev.org.uk/view/MOTR/job/MOTR_APP_loaddata/
457
458MOTR_APP_Loader_perftest
459This job is being used to run Subscription Loader performance tests.
460
461Script used: XXXXX
462
463Git URL: XXXXX
464
465Job URL: https://jenkins2.motdev.org.uk/view/MOTR/job/MOTR_APP_Loader_perftest/
466
467MOTR_APP_Notifier_perftest
468This job is being used to run Notifier performance tests.
469
470Script used: XXXXX
471
472Git URL: XXXXX
473
474Job URL: https://jenkins2.motdev.org.uk/view/MOTR/job/MOTR_APP_Notifier_perftest/
475
476MOTR_DEV_lull
477This job is being used to destroy development environments every evening in order to save money spent on the infrastructure.
478
479Jenkinsfile used: dev_lull.groovy
480
481
482
483Git URL: https://gitlab.motdev.org.uk/webops/motr/blob/master/jenkinsfiles/dev_lull.groovy
484
485Job URL: https://jenkins2.motdev.org.uk/view/MOTR/job/MOTR_DEV_lull/
486
487MOTR_MGMT_terraform_run
488This job is being used to run terraform against the MOTR MGMT environment.
489
490
491
492Git URL: https://gitlab.motdev.org.uk/webops/motr/blob/master/jenkinsfiles/mgmt_terraform.groovy
493
494Job URL: https://jenkins2.motdev.org.uk/view/MOTR/job/MOTR_MGMT_terraform_run/
495
496MOTR_NFT_JMeter
497This job is being used to run NFT tests against the MOTR NFT environment. For details refer to: MOTR NFT - Running In-Sprint
498
499Job URL: https://jenkins.motdev.org.uk/job/MOTR_NFT_JMeter/
500
501MOTR_security_burp_tests
502This job is being used to run security tests. For details refer to: MOTR 1.0 Security Audit
503
504Job URL: https://jenkins.motdev.org.uk/job/MOTR_security_burp_tests/
505
506 
507
508HOW-TO's
509Create a new live environment in the AWS Live account
510Let's say we want to spin up a production environment for the first time. This environment should be fully accessible from the Internet (= no WAF) using 'https://www.reminders.mot-testing.service.gov.uk' address.
511
512For the sake of keeping this how-to simple it does not contain any information about the capacity tweaks for DynamoDB and SQS.
513
514Create a new branch, ex. 'new-env-prod-mgmt' that will reflect changes that have to be made in the MGMT environment.
515
516webops/motr|master> $ git pull
517webops/motr|master> $ git checkout -b new-env-prod-mgmt
518Add 'prod' environment to the motr_environments list variable in the etc/env_eu-west-1_mgmt.tfvars file.
519
520webops/motr|new-env-prod-mgmt> $ cd etc
521webops/motr/etc|new-env-prod-mgmt> $ vim env_eu-west-1_mgmt.tfvars
522 
523[...]
524motr_environments = ["pp","prod"]
525[...]
526Merge  'new-env-prod-mgmt' branch to master and run https://jenkins2.motdev.org.uk/view/MOTR/job/MOTR_MGMT_terraform_run/.
527Log in to the AWS Management console and:
528obtain the KMS key ARN for 'prod' environment,
529request and approve a new certificate for 'www.reminders.mot-testing.service.gov.uk'.
530Create a new branch, ex. 'new-env-prod'.
531
532webops/motr|master> $ git pull
533webops/motr|master> $ git checkout -b new-env-prod
534Create an environment-specific tfvars file (for example by copying already existing one for another environment).
535
536webops/motr|master> $ cd etc
537webops/motr/etc|new-env-prod> $ cp env_eu-west-1_demo.tfvars env_eu-west-1_prod.tfvars
538Adjust variables values in the env_eu-west-1_prod.tfvars file, for example:
539
540webops/motr/etc|new-env-prod> $ vim env_eu-west-1_prod.tfvars
541 
542[...]
543environment = "prod"
544public_dns_domain = "reminders.mot-testing.service.gov.uk"
545alias_record_name = "www"
546cf_apig_channel_key = "<should be left empty for now>"
547base_url = "https://www.reminders.mot-testing.service.gov.uk"
548mot_test_reminder_info_api_uri = "<has to be obtained from the MOTH Trade API side>"
549mot_test_reminder_info_token = "<has to be obtained from the MOTH Trade API side and encrypted with the KMS key>"
550gov_notify_api_token = "<has to be obtained from GOV.UK Notify and encrypted with the KMS key>"
551kms_key_arn = "<computed - see pt. 4a>"
552[...]
553Reconfigure the https://jenkins2.motdev.org.uk/view/MOTR/job/MOTR_APP_promote/ job and add 'prod' environment to the 'TO_ENV' list.
554Merge 'new-env-prod' branch to master and build https://jenkins2.motdev.org.uk/view/MOTR/job/MOTR_APP_promote/ and choose appropriate values for the promotion process:
555 
556Go to the AWS console (https://eu-west-1.console.aws.amazon.com/apigateway/home?region=eu-west-1#/api-keys):
557obtain the value of the 'motr-web-prod-key' by clicking the 'Show' link next to the API key property,
558*** aws_api_gateway_api_key resource does not return key value so it cannot be passed directly in the to the aws_cloudfront_distribution resource
559link the api key with the 'motr-web-prod-up' usage plan.
560*** aws_api_gateway_api_key resource cannot be linked with an usage plan automatically
561Create another webops/motr branch or use the same one.
562Adjust the value of the 'cf_apig_channel_key' variable in the env_eu-west-1_prod.tfvars file by pasting the key value obtained from pt. 10a.
563
564webops/motr/etc|new-env-live> $ vim env_eu-west-1_prod.tfvars
565  
566[...]
567cf_apig_channel_key = "<obtained value>"
568[...]
569Merge your branch to master.
570Reconfigure the https://jenkins2.motdev.org.uk/view/MOTR/job/MOTR_APP_terraform_run/ job and add 'prod' environment to the 'ENV' list.
571Build https://jenkins2.motdev.org.uk/view/MOTR/job/MOTR_APP_terraform_run/ as the only changes we expect is the API Gateway origin custom header x-api-key value.
572*NOTE: Using the MOTR_APP_promote build with exactly the same values like before in pt. 9. would have the same effect but it would revalidate all artefacts once more
573
574Open a browser, go to https://www.reminders.mot-testing.service.gov.uk and check if everything works fine.
575NOTE:
576
577The 'alias_record_name' variable value for any other environment but production will be the same as the 'environment' variable value.
578The webops/motr|components/motr_app/variables.tf file should be queried individually per environment for additional variables that should be set.
579Consider CloudFrontvs.S3Bucketorigin when creating an environment.
580Destroy a live environment
581Go to AWS console https://eu-west-1.console.aws.amazon.com/apigateway/home?region=eu-west-1#/usage-plans/, choose your plan and unlink it from the API stage it is linked to.
582*** The Usage Plan is being created using the 'aws_cloudformation_stack' resource because Terraform does not provide a dedicated resource, which makes it impossible to destroy both API Gateway and Usage Plan whenever they are linked.
583Use https://jenkins2.motdev.org.uk/view/MOTR/job/MOTR_APP_terraform_run/ to destroy the environment.
584Rollback PRD environment to previously deployed artefacts
585Let's say we have deployed some artefacts to PP environment and deployed them (by promotion) to PRD.
586
587The manifest version was: 20170322092429_125
588
589The previous manifest version used was: 20170321171027_23
590
591OPTION 1 - promote the previous manifest version
592
593Run https://jenkins2.motdev.org.uk/view/MOTR/job/MOTR_APP_promote/ job and specify '20170321171027_23' as MANIFEST_VERSION.
594
595OPTION 2 - rerun terraform (does not restore static assets automatically)
596
597Run https://jenkins2.motdev.org.uk/view/MOTR/job/MOTR_APP_terraform_run/ job and specify '20170321171027_23' as MANIFEST_VERSION.
598
599If static assets were changed which is very unlikely to happen you need to download the appropriate assets zip file from s3/assets_archive, unpack it and sync/copy downloaded folders with s3 bucket.
600
601~ $ mkdir rollback_assets
602~ $ cd rollback_assets
603~/rollback_assets $ aws s3 cp s3://uk.gov.dvsa.motr.prd/assets_archive/motr-web-static-20170223133150-85531f25f7ef854946a82ea128590bca0d0609bc.zip .
604~/rollback_assets $ ls
605motr-web-static-20170223133150-85531f25f7ef854946a82ea128590bca0d0609bc.zip
606~/rollback_assets $ unzip motr-web-static-20170223133150-85531f25f7ef854946a82ea128590bca0d0609bc.zip
607~/rollback_assets $ rm -f motr-web-static-20170223133150-85531f25f7ef854946a82ea128590bca0d0609bc.zip
608~/rollback_assets $ ls -l
609total 0
610drwxrwxr-x 21 user user 714 Feb 23 13:31 images
611drwxrwxr-x 12 user user 408 Feb 23 13:31 javascripts
612drwxrwxr-x 22 user user 748 Feb 23 13:31 stylesheets
613~/rollback_assets $ aws s3 sync ./ s3://uk.gov.dvsa.motr.prd/assets/ --cache-control 'public, max-age=31556952, must-revalidate'
614upload: javascripts/application.js to s3://uk.gov.dvsa.motr.prd/assets/javascripts/application.js
615*** Only files that were changed will get resynchronised.
616
617Encrypt/decrypt a key with KMS
618This is how keys/tokens should be encrypted/decrypted using KMS.
619
620Encrypt
621
622$ aws kms encrypt --key-id <key_id> --plaintext <file_or_text_to_encrypt> --output text
623output:
624
625AQECAHh2/l0C6dasoy9qdck8e0+2CBExJRJyHEFcO0D/QAAAMIwgaIb3DQEHBqCBsT[...]AYJKoZIhvcNAQcBasdSALjARBAx+hzaelOn8npy2VZgCARCAe55A== arn:aws:kms:<region>:<aws account id>:key/<key id>
626where:
627
628<key_id> - id of the kms key (can be obtained from the AWS console)
629<file_or_text_to_encrypt> - this will be either the binary file or a inline text
630
631Decrypt
632
633$ echo <encrypted-value> | base64 -D >> base64_value.txt
634$ aws kms decrypt --ciphertext-blob fileb://base64_value.txt --output text --query Plaintext | base64 -D
635output:
636
637wIBV7HXJFG4asdadwSAAD7uQ3OvnHlyY3LNlIY6
638DynamoDB backup/restore
639For eventual DynamoDB backup/restore process the following tool can be used: https://github.com/bchew/dynamodump.
640
641 
642
643Additional information
644Error page
645Error page is being served from the s3 bucket whenever any of error codes below will be returned by the application. The fallback mechanism is provided by the CloudFront distribution's Error Pages configuration.
646
647Physically, the error page can be found as a single 'service-unavailable.html' file in the dvsa/motr-webapp repo in the errorpages folder.
648
649
650
651- If the origin sends cache headers, CloudFront uses them, but squeezes the caching time to fit between 'min_ttl' and 'max_ttl'. The 'default_ttl' setting is not used.
652- If the origin doesn't send cache header, CloudFront uses 'default_ttl'. The 'min_ttl' and 'max_ttl' settings are not used.
653- Default caching time for errors is 5 minutes, but can be overridden by Custom Error Response configuration.
654- CloudFront never touches/adds/removes the cache headers that are passed to the client, so none of the above settings affect the clients.
655CloudFront vs. S3 Bucket origin
656All buckets have at least two REST endpoint hostnames. In eu-west-1, they are <example-bucket>.s3-eu-west-1.amazonaws.com and <example-bucket>.s3.amazonaws.com. The first one will be immediately valid when the bucket is created. The second one -- sometimes referred to as the "global endpoint" - which is the one CloudFront uses - will not, unless the bucket is in us-east-1. Over a period of minutes (experience proves it's ~ 1h40min), variable by location and other factors, it becomes globally accessible as well. Before that, the 307 redirect is returned. Hence, the bucket was not ready.
657
658The implication of it is that you can get the following errors when accessing your newly created environment:
659
660
661
662
663
664Because caching mechanisms are being used, the only confirmed way to get rid of this errors and get assets being displayed correctly is to invalidate /assets/* on the CloudFront level. However, there is another option which requires creating the skeleton of an environment in two steps:
665
6661st step: run terraform with 'with_cloudfront' variable set to false
667wait until "curl -Ik https://uk.gov.dvsa.motr.pp.s3.amazonaws.com" is not returning 307 anymore
668
6692nd step: set 'with_cloudfront' variable to true along with any other variables needed and run terraform again