Zzm5iDFC

1oG	This is the name of the game: City of games
2Office Macs and Dev Machines	
3This is Ringzero.
4
5Macs: We use Macs in order to build for iOS. One of the mac mini is the master and tells the slaves what to do. Jenkins is installed on all 3 macs. There is no human interaction here. Once there is a commit to GIT, the master will tell one of the slave to build. Once it is built it should upload automatically to Hockeyapp or Testflight for testers. In the future it is planned that the mac mini are used to build Android. Right now developers do it by hands on the dev machines. Those builds are uploaded manually to Hockey app for testing or Apple store for releasing. The macs are not accessible from outside. The macs are supposed to send assets automatically to S3 but in reality this feature is not used and assets are sent manually to S3 with Skipper.
6
7Dev Machine: they are used for the game development (Unity..)
8
9Skiper Web Layer	
10Skiper runs on NodeJS and AngularJS 1. It provides a backoffice to configure the game system (popups, turn on or off the slots, setup redemption items that is now partially replaced by the leap store). You can get information about the player (if they are banned, their current cheaps, their current reward points, devices they are using, their past redemption). If someone complains we can give chips back by using player ID (unique ID generated by CoG). Reports can also by obtained from here. There are 2 types of reports. There is an automated report for redemption (it does not work well because it goes through 5 steps and if one has problem it does not really go well) and the other type implemented for Melco (data analytics create those reports and upload them to S3 and they are available in Skiper). Assets: you can upload assets here: it can images, text or unity assets. They are localized and versioned. There are 2 machines of Skipper Nemo (production) and Dory (development). They are both able to talk to all environment. But they are only hosted separated from other environments. They use their own security layer to access dev, QA, stage and production. You don't need VPN to access Skipper. This is the URL to access it: https://skipper.goldenratstud.io
11
12To go on Skipper you need login/password. Skipper uses an encoded token to communicate with gatekeeper (turn off slots..). It can connect to analytics database (somehow).
13
14
15
16The logins for skipper... are stored in Web user and config MangoBD, but the game data is stored in Shard 0 through gate keeper.
17
18Web user and config	This is an instance that runs MangoDB and that is used by skipper.
19GIT Hub	This is the cat. This is code repository from frontend, backend and analytics scripts. This is hosted online (third party).
20Docker HUB	This is the whale. This is also a third party service. When you build an image of a backend service like Gatekeeper, Bookeeper, Columbus, Quest system, (everything in the blue box in the diagram). Dev machines can get latest backend builds from Docker Hub.
21AWS	This is Amazon Web Services. The access is restricted by security group (IP range). You need OpenVPN to access all the machines in AWS. It is an EC2 instance (Ubuntu). The development, QA and stage are hosted in US West. And the production environment is hosted in Singapore.
22OpenVPN	The purpose of this server is to give access to other servers when connected to it. It can only be accessed by set IP ranges (from Ringzero). We can run OpenVPN or another tool such the one we use in house sophos ssl vpn client to connect to AWS with OpenVPN. There is only one OpenVPN server that can talk to all regions. OpenVPN
23Backend Master	
24It runs Jenkins on Ubuntu. It is only accessible via VPN (in going) but it can connect to outside to GitHUB and DockerHub for example. When users is connected with VPN to OpenVPN, it can access the Jenkins page by using this URL: http://build.goldenratstud.io/
25
26Jenkins is used to automate building and creating Docker images that are put on Docker Hub. It is currently done manually by developers who should only use it for QA and dev builds. Productions builds should be done by DevOps. There are 3 option Build,.. Select the third option that does build but also create the docker images. The user must also select the images it wants to create (e.g. columbus, gatekeeper, route66, socialservice, workers, messenger, league, bookkeeper, leaderboard, tournament, statusmonitor, turbine, quest, fengshui, slotmachine..).
27
28The backend master is supposed to do backups of Cassandra but this apparently it does not always work (every 6 times it works). We need to look at the script to know where the backup is going.
29
30Backend Master informs Kubernetes about new Docker images available in Docker Hub.
31
32Backend Master was supposed to send reports to Melco via email. This is not used at all and it is replaced with PowerBI. It was previously connecting to Analytics Data.
33
34It tells Spark to do his job.
35
36It does not deploy analytics reports anymore and and Thomas Analytics visualization is not used anymore.
37
38
39
40Jenkins runs 3 jobs (batches):
41
42DAU aggregate (daily: checkout a shell script from GIT with takes a jar and combines with another jar file stored in Jenkins server and it submits it to spark and copies the file on a specific folder and then tells spark to start the job),
43
44Payout aggregate (daily)
45
46Analytics main, it is supposed to run all the time: if the job is completed (jenkins cheks this) then it starts it again. This scripts runs on Spark. This takes a lots of messages from Kafka (not all like hadoop) and put them in analytics database with not much processing. The reason is that data scientist are more confortable to writte SQL querries than Hadoop.
47
48Kubernetes Master	It runs Kubernetes on Ubuntu. It needs to be accessed by SSH. It takes docker images and deploy them. It also manages nodes and take actions when needed (e.g. performances issues..).
49NGINX	It is a web proxy/server here used for load balancing and for SSL (and probably some security DDOS attacks). It is in front of Skipper, gate keeper..
50memcached	All servers that use Spring boot (almost all servers) are using memcached. Memcached is used to have result of queries for example getting the number of chips of a particular user. If the result is not in memcached, it will query the database and put it in memcached. When writing to the database, we need to make sure to update memcached at the same time. It is mainly (only?) used for player data (not for master DB).
51Redis	Redis is used for the 15 minutes tournaments for slots (for all the slots). Even third party vendors can join tournaments. We could use only Redis instead of memcached, or memcached instead of Memcached. [This could be a technology optimization].
52Leaderboard	Leaderboards work with tournaments. Leaderboards are stored in Redis. It communicates with Redis. It is temporary data. All the instances that run Springboot are run on Centos. Server is probably tomcat.
53Tournaments	Tournaments work with Leaderboards. Tournaments are only for slots. 
54Smartfox	
55Smartfox is a game server that you can host. https://www.smartfoxserver.com/overview 
56
57Smartfox is also a framework on which is built Baccarat. Baccarat is a plugin for Smartfox.
58
59It manages the baccarat rooms. It manages the content of the rooms. There are about 100 rooms depending a number of users. There are no private rooms. You choose the range you want to bet. All players will bet within the same range in the same room. The maximum people is 12 (?). Gate keeper asks Route 66 (name of a micro service developed for COG) for the list of the rooms. 
60
61When Smartfox starts it starts a minimum amount of rooms, roughly 10, it will request from the bots server (Baccarat Bot server) some bots to fill the room (3 or 5). It will publish a message about the room via Kafka. Route 66 consume the message and keeps track of the available rooms, it knows the IP and room numbers.
62
63Smartfox takes care of everything in the room. Smartfox has the logic of the game for the baccarat. It does the dealer job.
64
65there are multiple Smartfox servers running.
66
67Smartfox has sometimes to deal directly with Gatekeeper to get the number of chips. But there is bug. Smartfox and Gatekeepers know the chips amount of the player. There are 2 cases where money is complicated. If Julio puts money on the table, for Smartfox assumes the player does not have this money anymore, while Gatekeeper will assume that Julio has all the money. So one of the service crashes it can lead to chips inconsistency.
68
69Baccarat Bot server	It runs Java Springboot. It simulates clients playing (bots).
70Route66	It is micro serviced developed for COG (not be confused with AWS service which is route 53). It is used to find active Baccarat rooms. Smarfox publish to Kafka and Route 66 records the rooms. 
71Quest services	In the game you have quest and achievements used for baccarat and slots. We are currently working on this micro service. It keeps track if you complete specific quest and keep track of progression. It stores data in Volatile data.
72Volatile data	This is a Cassandra database. It has about 6 servers in a cluster. It is more "volatile" than player data but yet it is player data. We store here quest progression and daily login bonus. This is where will be stored new quests data. Book keeper might also store data in Cassandra.
73Bookkeeper Reward booking	Bookkeeper is responsible for rewards booking. It also generates the QR codes. Gate keeper directly communicates with Bookkepper, it does not uses Kafka. All things does in this system generates analytics events, so it could use Kafka here.
74Kafka - message broker	It is a message broker. It is mainly used for analytics. Right now it is an EC2 instance but it could be migrated to AWS service.
75Temp data storage Hadoop	
76It stores all the messages that are in Kafka and it never delete them. Hadoop is used for big data. Right now it is about 150 GB. There was an issue with Hadoop that needed reboot everyday. There is an automated script to reboot it when it gets stuck. When it goes here it tells Kafka that it consumed the message anymore. So if no other services registered to get those messages.. In Kafka, messages time out after 10 days. So if you have a failure for more than 10 days you can loose data.
77
78On Haddop you see a Pacman this Apache Gobblin. It is a script that connects to various data source and store them in Hadoop. It takes the messages from Kafka, add a timestamp to the message on it and puts it into Hadoop.
79
80There is a daily backup of Hadoop (supposedly) on S3. But note that it overwrites the previous backup. This is something to look into.
81
82Spark Execute analytics	It is is cluster of servers that distribute jobs. Sparks is only uses to process analytics. It processes data stored in Temp data storage (Big storage data) and it stores the results in Analytics database. Jenkins launch the job. They wanted to do their own analytics tools, and for reduction of cost did not use tableau.com (to replace spark), or flurry.
83Analytics Data	This is where is stored the results of analytics queries. This is a Postgresql database.
84Player Data	
85This is where the player data is stored (in addition to Cassandra).
86
87Right now there are 8 Shards. They store user data: balance, rewards,.. It uses Sharding. The shard id is in the player id. The database can generate new player ids:
88
89create sequence if not exists global_id_sequence;
90CREATE OR REPLACE FUNCTION next_gid(OUT result bigint) AS $$ DECLARE
91our_epoch bigint := 1442793600000;
92seq_id bigint;
93now_millis bigint;
94-- the id of this DB shard, must be set for each
95-- schema shard you have - you could pass this as a parameter too
96shard_id int := __SHARD_ID__;
97BEGIN
98SELECT nextval('global_id_sequence') % 1024 INTO seq_id;
99SELECT FLOOR(EXTRACT(EPOCH FROM clock_timestamp()) * 1000) INTO now_millis;
100result := (now_millis - our_epoch) << 23;
101result := result | (shard_id << 10);
102result := result | (seq_id);
103END;
104$$ LANGUAGE PLPGSQL;
105
106This is executed in Postgres shard directly (any can do but probably uses number 0).
107
108The exception: shard 0 also contains the data of RMS (data to configure items) in addition to player data. RMS data is configuration data of the game set in Skipper.
109
110RMS	
111RMS has also some configurations files identical to what is in Master DB.
112
113Master DB	It stores configuration of the game. It differs from web user and config database that only stores skippers configuration. Web user and config database is not used by the game while Master DB is used by the game.
114Analytics service for Power BI	This is a service for statistics. A server at Ringzero gets analytics data (via OpenVPN) and sends it to Power BI Office 365.
115Facebook - Social services	It retrieves data from Facebook: for example it checks the token, retrieve friends.
116Columbus	This is something from Netflix library that programmers called Columbus. Eureka is the library used. This is a discovery service for the whole application. All the pots when they start call Columbus. They also ask Columbus where is config service. They then ask config service their configuration. Columbus DNS is configured in Kubernetes. It knows what is where and it answers to queries to where is x service. Columbus is also the load balancer.
117Config service	Config service has the configurations for each micro service: bookeper,... It connects to Github when it starts and download the configuration files. When changes are done of Github it is unsure how Config service gets updated. Maybe we need to restart it.
118Baccarat	This is a game “mode” in CoG. Other “mode” is Slots
119S3	This is bucket (storage) used to distribute assets for CoG: images (from Skipper), assets bundles
120External slots	External are HTML5 games that are loaded from the client. Gatekeeper is dealing with them directly.  We have a list of slots that the player can choose from. These are URLs. The client sends a token to the slots vendors. (There are 3  player token: the COG token it is unique per device and token, the third party token for slots, and leaps token. On login gatekeeper gives the 3 token). The slots vendor can ask gatekeeper how many chips the person has and how many chips he can bet. When you press spin in the slot machine: our webview sees that there is a request to the slot vendor and ask gatekeeper about the result by using the COG token; the slot vendor will also request gatekeeper with the Third party token, it will send the amount bet and the result. Gatekeeper believes the amount won from slot vendor, it will only check the amount bet. When the slots connect to Gatekeeper they use API key and third party token (which is a player token). The API key is per vendor. The client connects regularly to gatekeeper to see if there is update in order to update the UI around the html view.
121Internal slots	We have dedicated pots to handle slots. The request comes from the client and go to gatekeeper by using COG token and asks for the result.
122Messaging	This is not a chating system. This is not Kafka. It is a socket that keeps a tunnel/connection always opened so the server can send requests to the client.We send when a friend joins a baccarat table, we send when a jackpot is won. It is responsible to send events like friend is in lobby, friend joint a specific slot or Baccarat room. I also has the functionality to send a mass notification to everyone
123Gatekeeper	
124api endpoint for all games. It makes sure you have a proper account, your balance is correct and forward the response to the service id. Everything except from Smartfox and messaging system will go through gatekeeper. All request except for the Baccarat game and part of the slots go through this api.
125
126Leaps integration is done here.
127
128Route 53	It is used to manage DNS.
129Monitoring services	We have 3 different monitoring services. New Relic monitors the state of the machines. It is not hosted. The docker images have a service on it that send data to New Relic. Zabbix does basically the same thing: there is a server for it in EC2: it is part to the security. To access to it you need to use VPN. There is also Nagios also does the same thing.
130Feign and Hystrix	
131Those are libraries that we are using. Feign maps json to java objects. When the client send s a request, it sends it in json format. When the server sends a response, it sends it in json format too. What the library does it is instantiating an object that must have a definition with the json data as parameters values, it can also create dictionaries.
132
133Hystrix is to handle transaction between micro service. This a library from Netflix. It is a transaction management for micro services. For example if gate keeper receive one request from the client, it might have to do 2 requests to different micro services: one might work and one might fail. In case of failure Hystrix could revert the transaction (or not commit it).
134
135Amazon cloud front	When we put data on S3 Amazon will distribute the data globally so you can deliver the data from the server closer to you. The problem is that the data is not available immediately, you have sometimes to wait for 10 minutes.
136Workers	Have multiple asynchronous functionality like password reset, …
137League	is for league play in DDZ