System Architecture for Web Application

all right so this is a video for understanding how to design a system especially web applications but the concepts mentioned in this video are not only applicable for designing web applications by an 80 type of software but the processes and the steps will be especially targeted for web application so what I'm going to be sharing is based on my experience in this world of how to build software there are a lot of different opinions and a lot of different ways of building a good software so I'm gonna try to lay out some general principles that

I think are important but do understand that this is based on my experience and other people would have different opinions about how things should be architected but I'm hoping that me sharing this will help you understand how I would go about in building web application a complex web application and what technologies I would use what programming languages is I would use which frameworks I would use what services I would use and what type of questions I would ask this could be especially helpful if you are working as a contractor and you're trying to explain to

a potential client how you will go about building the system because they maybe want to know how you how you would go about and they may ask you well why are you using Python versus Ruby or versus node versus JavaScript you know or something like that so can also be helpful if you are getting ready for technical interviews and the hiring manager asks you if you're building an application that has X Y & Z how would you go about building that okay so it's applicable for all of that so depending on where you come from

it will probably have different lessons for you okay so I'm gonna be sharing steps that I follow this took me a while to document and I actually didn't even know whether it was possible to document my workflow like this but I tried and I think I came up with something that's quite unique so I hope you like it alright so let's go right to the document so this is my process I was trying to think when when someone asked me Michael how would you build an application that does ABC well usually I have some strong

opinions about one language you should use what framework you should what you should not do I have some stronger opinion about that but how do I actually come up with those insights and is there way that I can actually create a process diagram that anyone can follow to leverage my years of enterprise web application development experience and initially I thought no it's not possible it's just based on bunch of experiences and insights and it's intuition but once I started writing down well let me see if I can actually do this to help people I was

surprised and I was able to do it so I think this is basically my 12 years of no house and knowledge about system architecture in five pages no please that I'm learning there are all these new tools that are coming out you know I don't know everything you know there are some things that I know but a lot of these things that are coming out right now especially technologies that are going through a lot of innovations I have to constantly learn so you know you may have some developers with more experience than me on some

of those those parts that have innovation that you can also learn from so we're all learners here in life so let's not try to fight over who is right and who is wrong but try to understand each other's point of view and learn and you know hopefully a lot of these things can be internalized so let's say in this hypothetical scenario your client or supervisor wants you to design how you would build a complex web app or it could be your potential supervisor who wants to hire you and they say hey I have this application

how would you go about building it well the wrong thing to do is spit out answers never spit out answers right away you know that if you spit out answers just right away it's like imagine you meet with me and I say hey I want you to build a house it's a five bedroom house in a three bath and if you just go and build that house right away like what's going on or you imagine even worse I say yeah I want to build a simple simple house that's five bedroom three bath how much is

it gonna cost how long is it gonna take are you gonna give me the answer right away and say Michael it's gonna I can do it in two years or one year and it's gonna cost you a half a million dollar now I'll be like oh that's a good deal because what I'm thinking is I'm thinking about a 15,000 square foot mansion with five huge rooms and three huge bathrooms that's like the size a swimming pool and you quoted me half a million dollars all right I got a sweet deal and you said you're gonna

build it and how long six months one month or one here all right okay you're hooked and if you don't build it in six months or you're you're screwed if you are if you need more than half a million dollar then well I'll refer to the contract that you sign right I won't do that but it's some of your potential customers may write never run into that trap when they are saying is a simple house and to a client is always simple they may not know how to build it but to them is always simple

right and it may be simple in their mind but in actually to build it it may not be so simple right I'm not looking down at them right because again they don't understand what the efforts that needs to go inside maybe the idea is simple right and but it requires a very complex implementation but anyway what they are thinking in their head about a three-bedroom house or five bedroom house if what you're thinking is not going to be the same the way you do it is by asking questions listen before jumping in ask a lot

of questions and don't assume that's one of the key principle okay so I tried to put the key principle in the yellow bucket and how you do it I put some tips in blue right why we do things is always always more important than how we do things although both combined is what makes things very powerful right if you don't know why you're doing things but you just follow steps then you become a typist you're not a developer you're a typist if you are if you only know why things should be done but you don't

know how to do it then maybe you can be a good manager but it will limit your ability to create things so you need both the why all right the principles behind it as well as how to do things so if I was in this stage I would ask questions to better understand all right some of the questions you can ask could be hey hey how many developers will be working on this how many front end back ten because some of these you will find out that a lot of your process will depend on your

existing team you can ask them what are the languages and frameworks these four developers already know that's important because to learn another language and framework will take long time sometimes if you have crappy documentation that you're going through and you don't have the mentorship that you need it could take maybe a year to three years for you to get familiar with one language and framework right you don't want to go through all that time so if find out who will be working on this right or am I the only person working on it right what

part of this you can ask them even questions such as what part of this application do you think will get the most traffic will have the most bottleneck you can ask them they would have some ideas how important and this is this is a good question I think about how important is speed a development versus stability of the project or stability of the application do you if you all could only pick one not ball right where one we're building things fast but we have bugs along the way we catch them but there are bugs along

the way but we value speed to be a lot more important than stability then we go one round if they say no we can't tolerate any bug I would sacrifice how fast we build things to have a more stable version then it will also affect your system design right ask them some of those questions in advance so you can you can choose they will always want both but you cannot have both right ask them which one would you would you go with and there are some other questions that you can ask you can ask them

are there other api's or micro services that we can utilize that you know and this is some what some of these are good questions that you can even post during interviews how would you design something something something well which part of this application would you anticipate the most traffic and bottleneck and why right worst thing to do when you're going through interview is not to ask questions and just assume ask them questions dialogue okay any specific person's preferences on which programming language framework or test framework because they may have some strong opinions some of them

may say I don't care you pink that's fine if some of them say I really want you to build something and XYZ well I'll go ahead and put that into top okay step number one is that step number two research phase so don't go and just build things there may be some other libraries or modules that could or api's that could really make the job easy so go in and do some research maybe five ten hours of research to find out what is already available if somewhat already build some modules that you can utilize why

spend months building that yourself when you can just tap into that and use it especially if those are open source there is a trap that you can fall into you got a limit the amount of time you do research because some developers may spend too much time just researching and piecing together other people's code rather than actually building their own that is a dangerous trap and that is a trap that developers who don't feel quite comfortable with the core building blocks and then they're not at the point where they can actually create these frameworks and

libraries they tend to lean on other people's code that much but resist the temptation but also don't go overboard and develop everything from scratch yourself - especially if there are some strong libraries and modules plugins and ap is that you can use to to build this okay alright so if you're doing any much and learning things make sure you look into Python libraries to see if there are some libraries that you can leverage a lot of the machine learning algorithms some of those things may be available through and why train your dataset and create new

you quote for you when you can just leverage existing code and data that's available out there right so do the research step three is you choose a programming language and frameworks step four is then you choose the data where you're going to store the data step 5 is then you figure out how you're going to deploy in the cloud step six is how you're going to manage all the different services so now I'm going to go through more in-depth how I do step three four five and six so step three let's just walk through some

of it and this will be available and I'll try to update this as we go but let's just go through one by one and I'm gonna go slow intentionally so that you're not overwhelmed and we can cover some of the important steps so it's a passive way of learning the active way of learning is by you actually applying this into a real application and you know finding things that works and things that I may have put down here you're discovering that and adding it to your own workflow but for now passive mode is okay let's

go through step by step the first question I ask always is is there anything that needs to be done in real time so imagine like a drawing app live chat we're drawing out where people can collaborate and draw things together things like Google Doc where multiple people can collaborate PowerPoint you know where multiple people are creating slides well I don't know if that's really collaborative but real-time right real-time alert services is there anything if there is only build that component the real-time component is a separate app okay a separate app is all so we're just

going to call it a micro service those are interchangeable okay a small app that's built is especially for a specific purpose it's also called the micro service but it's just an app okay so only build a component using nodejs express and socket but why why why not build everything in Express so there may be some other frameworks and know that you can use Express is one of the dominant one but notice quite new there are our frameworks that are more mature such as rails and Django and others and they're meant to handle large complex applications

but Express is not an MVC framework although you could convert and be Express to be an MVC framework it's not hard to do but a lot of developers don't really use node for that they're using node form there is something very simple for a very modular service so it's best to just build that specific part of the application using node because node is nominal when it comes to handling real time especially with socket IO IO okay so build only that created as a micro service and connect the other apps through API which we'll talk a

little bit later about okay so that's important okay mistake big mistake is to build everything in one framework huge huge huge mistake and to build that giant framework just using node and expressed horrible mistake you will run into issues never do that right this again expresses a light framework light framework is not a mature framework that's built to handle lots of codes okay you can organize it but again this is you will probably run into issues it's better to use a mature framework for something that will get large and use light framework for building things

that are light right use things for what is built for then I ask is this a weekend project if it is a weekend project something that you can just put together you know in a week maybe a few weeks yeah you can use a light framework not a full-blown MVC framework so for example if you're using Python you something like flask it's not MVC you know you can do things procedural but it's a good framework and is it doesn't come with a heavyweight of most MVC framework so you use that if you're using Ruby you

can use the Natura instead of rails there are dozens of these light frameworks available for each of the programming language they all pretty much do the same thing there's you know there again convergent so that's what you would do and you'd be allowed and then another question I asked is well how many back in engineers will be working on this and the reason I that we I asked this is because this principal here having a small team of engineers per project is better if you were to graph the how efficient your software development is and

if you were to graph that with the number of engineers it's okay so it is again think about productivity on the y-axis and then the number of engineers as you put in more engineers is the productivity going to go up in a linear scale so if the productivity will was tool when you had two engineers if you have four engineers is it a Productivity gonna be four no the way that works is actually kind of goes in a linear fashion and then it kind of tapers off and then it starts actually going down as you

put more engineers so what is the ideal number of engineers back in engineers you should put on the project different people would have different opinion some there's a pizza rule where you know never have the bigger team in your bigger engineering team then who you can feed with the single pizza why pizza pie but that depends on the country too and you know if you if you live in a country where they eat a lot maybe a bet the answer will depend right so my rule of thumb is you know five to ten I think

it's a good size 10 is kind of you're going to the large number five that's good all right you can go have lunch together you can talk to each other but the communication number of steps needed for communication goes exponentially as you is the team size grows you know we want the small team and a small team to do some powerful things some of the you know world's really awesome software were built this by half a dozen engineers working on it that you will be impressed you can do a lot right so the idea is

have a small team of engineers per project to increase efficiency so if you need more than ten engineers then is simple find a way to break it find a way to break that app into multiple microservices or multiple apps so for example if you're building something like turbo tax online he's at one giant app and should be builded in Ruby Python oh you know and you put hundreds of engineers working on it no way you want to break it into microservices so you may have one yeah that just handles for example the website what people

when they go to a TurboTax Online comm the website they say that that's all it does very simple maybe one just you know goes talk to the county records maybe specific County records and just retrieve some of the tax information available for that county and that's all it does another app would maybe just do the computation for the state of California another app micro service will do something else for another state maybe you have another micro service that talks to these two other micro services and just coordinates information in between right you can almost always

look at a giant system and figure out a way to break it into pieces what does that do that makes this event I have a small team of engineers per project it's efficient teamwork is there no one's overriding someone else's code by mistake and things are a lot better okay so you break it down and then for each of the project now for each of the micro service you ask which one is more important speed or stability and they say hey we have zero customers right now we gotta build things fast fast fast and launch

what do you do well speed is more important so maybe you don't worry so much about test cases you just write things right if a stability is important because you already have lots of customers you can afford mistakes find a framework or find a language that come with some strong test cases I personally like r-spec and capybara a lot there are some other frameworks testing frameworks that are really cool and again even if one language is cut was tend to detent tended to be weaker and testing it will get better over time all of these

technologies are kind of merging converging so pick up pick one that you know that has a good test framework if stability is really important right if not then you don't really need a language then it comes with the strong test framework right especially after later you can just add you know some of the some of the integrated integration tests later on okay so once you know speed is important that you just don't build it I I do ask you know doesn't require any machine that learning or any like heavy math computation statistical analysis that's the

case Python has a lot of you know create libraries because Python was used a lot by scientists and engineers so scientists naturally they needed to do a lot of math so they built a lot of Python libraries for math and this is again why Python is such a popular language for machine learning because machine learning is just a lot of computational math well in a nutshell alright so python is crying because it already has all these libraries that Python scientists build so that's the reason why python python is the popular language for machine learning anybody

do that okay so now let's see does it mean a lot of math computation then yeah used like Python if not you can then what framework you use I mean use an MVC framework as the foundation I really think they're all similar anyway if you are pretty new then use a easier framework to learn such as well Django is not so easy but it's not too hard if you're doing PHP pick anything like larval coding errors and if you're doing Java you can do spring or some other things Java and c-sharp will have a little

bit higher learning curve than some of the other languages begin most of the MVC frameworks are kind of converging they're all becoming really similar to each other so I don't think it wouldn't matter and then once you design these projects then you create API calls so that they can talk to each other okay all right so that's the principle once you have a lot of different micro-services then you can either build your own API or later on I'll introduce some other services created by other people that you can use to connect some of these apps

to communicate with each other some of those other service letters I'll introduce a little bit later okay but the key principle is have a small team connect multiple maps using API and then when you do this you're utilizing strengths of each language and framework now if you have a lot of different experience with multiple frameworks then you will have some strong reference for example you know I have some strong opinions about how Django or M is done and I have some strong opinions about how what I like and dislike about multiple frameworks but I built

that over the years because I've used them all well all the ones that I know and maybe half a dozen to a dozen framework so if you don't have a strong preference just cool with what you're comfortable with but if you happen to have used a lot of different frameworks than just levers you know your preference and you know pick the one that has particular strings that you want to you want to do okay all right another thing to remember is don't over engineer you can also over engineer on the test cases and sometimes to

build one feature you're writing too many test cases and it really slows things down so you also want to have a fine find a good balance on how many tests you write and how quickly you implement the features that's really important so this is the back end what about the front end well does your app need to be a single page a lot of developers love single page they love they love learning new technology so it's you know they want to build something using the latest react or angular or a JavaScript library to create a

single page application because they want to learn cool technologies other than just using what they are familiar with and I understand that but you should really ask well if we were building things for the client we gotta ask does it really need to be a single page a lot of times it doesn't need to be so if it doesn't need to be hey don't build it it's a single page and my rule of thumb is does for each of the route the URL how many JavaScript lines would it need to handle the interaction and if

I think you know a few thousand lines okay maybe you don't actually need a heavy framework like react and angular and of course that depends on preference that's just my personal kind of gauge to see well do I need to just build it using plain JavaScript and use my knowledge and old P to organize my JavaScript code and maybe create a light framework or should I you know use a heavy framework like you know react and angular so you can you know do that okay so again the key thing is don't over engineering you need

you're gonna have this maybe a few hundred lines of JavaScript and you're gonna use react mmm why there's gonna be a learning curve if you use those heavy frameworks and then it will just slow things down yes it will you know scratch your curiosity but you know speed is also important or for the business and the customer can say remember that all right step four alright so now you know we've figured out which language which framework we use where there is going to be one one app or a multiple micro-services all right you broke that

down and of course all of this you've gotta practice you're going to build some real applications and really put this into use well for each of the app or each of the microservice you also have to think about where am I going to store the data what is my process Michael choice process so then I ask any data large files long text blob that could be stored in services like Amazon s3 the rule is do not store that in your database do not store a large file into your database can you store a large file

into your sequel database yes you can but do not it will slow things down it will make your database too big keep your database small store those files in Amazon s3 or some other services that you like and simply store the link to that file that will make sure that your database stays small you want a small database all right so you know that's really really important and then you ask yourself is that is any data visit does it need to be stored permanently for years for months if the answer is yes then you look

one way if you're on like handling doing things that are in real time maybe information that you got five seconds ago once it's been delivered maybe you don't leave to store it if you don't need to store it you can consider just using a memory based database such as Redis memcache and other things there probably dozens half a dozen to a dozen maybe more memory based database that you can look and you can use those because memory based database is really quick too and now a memory is really cheap compared to before so you can

store a lot of things in memory and compared to before gigabytes and gigabytes so you can consider that but if it does need to be stored then the question I ask is do you need to store do you need to do a relational or non relational database okay that one is a little bit tricky let me let me see if I can user and use an example here okay so I switched over to my iPad with the Apple pencil how do you tell whether you need a relational database for non relational database let me see

if I can explain that just very very quickly or as quickly as I can the way to think about that and the analogy that used is maybe there is a banette better analogy but let me try this one is let's say you decide to store information on a piece of paper and you're gonna store that in a cabinet drawer right our database ask yourself what type of record do you want to store into this cabinet so you're going to insert a paper but how do you want this information to look like relational is where the

field types are specified so imagine you get this empty form and it says fill in your name fill in your age fill in your city and fill in some other things right if these are filled and there's an empty space that you want to fill out so you pull out this empty record you fill out the blank and you want to store that into the cabinet then you want to go relational right if you want these fields predefined non-relational is more open and it can be like the left side is also blank fill in whatever

you want that's non-relational you might be thinking what okay maybe I confused you more let me give you an example imagine that I wanted you to create a web application to store any information you can find from anyone in the world relational approaches okay I'm gonna find out their name age City birthdate and blah blah blah you can imagine how that could work but maybe oh you come to me I'm managing this cabin and they say hey Michael you know I need to fill out this information for this new person in the world may how

long do you think that paper that I'm going to give the u.s. maybe 50 pages maybe hundreds of pages because every single thing that you might want to capture like hobbies you know their first boyfriend/girlfriend you know like their favorite dessert all of that would be a field on the left and then so a lot of that information a lot of that information right you may just leave it empty right like maybe they don't want to tell you what their hobbies are so you skip that part right but you can imagine it could be pages

and pages all right so that's the relational Pro sometimes this works and most of the data is stored in relational but if you don't want that and you just want to say Michael just give me a piece of paper and let me write whatever I want and store whatever I want then you can do that so for that particular example the non-relational has some benefit right because you can just say oh they wanted me to tell they wanted meat they told me about hobbies so let me write down what their hobbies are the downside of

non-relational is can you imagine another worker may have put hobby instead of hobbies so now you got all these data that is similar but you got to kind of clean up the data yourself that could be a problem with non-relational database because developers may store information all a little bit differently and to clean that up could be a nightmare right but it has the flexibility usually with any pros and strengths comes this disadvantage right so those are some of the trade-offs so again if you're building a web application you can say do I want to

give my workers a preformed filled on the left on what fields I want them to record and then send it to the database or you won't want to just give them a blank sheet okay the and I think that will kind of give you a good idea on whether you want to use a relational or non-relational database so that's usually what I ask and if it is a relational database then I ask myself how big will the data get in the next three to five years and this is a very soft rule of thumb and

technology gets better so this limit will change to at the time of this video Amazon relational database will handle data up to 16 terabytes that's huge huge huge but depending on I mean if you're logging everything that's happening in the world I mean you will need a lot of data storage but most of the web applications will be fine with one terabyte or less or a significant less so you can use my sequel Postgres mario to be or one of them there are slight differences the databases you shouldn't use are things that are used for

more development and testing or for people to level up so I wouldn't use sequel line but you know my sequel Postgres Mario DB Cassandra all these other things they all work pretty well but it does hit a limit when you get into the certain size usually by the size of the hard disk hard drive so if the hard drive later comes out in a computer and it has 100 terabytes then you know your database can probably handle up to that but right now most of the hard drives are like one terabyte so that's the limit

if you need something more you can go into Amazon RDS every single cloud services will offer these as a service that you can just tap into if you need something more than that you can use Oracle you need something way more than that useful do big data again you can set up Oracle servers yourself you can set up Hadoop servers yourself or Amazon Microsoft Google cloud already set it up and you can just use their service they made things a lot easier before they have the services you have to go learn it build it and

then deploy it manage it all yourself but now there are platform as-a-service then you can use to do that okay same same rule applies to non-relational you can use something like MongoDB if you're doing non-relational why would you be maybe able to handle a lot more so I always look up to the documentation to see what it can hand but if you need something way bigger then you don't rely on Amazon AWS or look on some other services to see if he can you know leverage that I'll put some links so you can find out

from Amazon what different database products they offer alright so this is the thought process database is usually not that hard it's pretty straightforward alright let's talk about the next step choosing your testing deployment process for each app or micro service so the first way is you set up the github can be used as a collaboration tool there are so many ways that you can use git in a lot of different ways but the way that I do it personally is I would create a github for each repository for each of the project that I'm building

ok and then I ask myself does this need to support multiple versions of your products so for example if you're creating some desktop app you need to support version 1.9 2.1 2.4 2.6 then you go into this bucket if you don't because this is one version that you need to support and you're not it's a relatively simple project what do I mean maybe it doesn't really require like five plus full-time back-end engineers to work on projects for months then you follow this bucket okay and if you if it is a simpler project and you only

need to maintain one version use what's called a good lab flow okay you can read a little bit more about this it's great for managing simple projects and you can read more information in the links over here if you're the clump the project is more complex you use what's called a gift flow and you have one repository where the source code is there and then you don't have the developers work on that repository as collaborators what you do is you have been for that repository you create what's called a separate branch and then that's where

different people work on that separate branch and they submit a pull request to the to the to the main repository or the original repository this will take some time for you to get familiar but most teams I would say follow these two approach at least the companies that I'm aware of in Silicon Valley so and there are some additional links here on the bottom where you can learn more about these different branching technologies now when you do this what you can do is you can set up hooks so whatever certain events happen in your again

you can have it trigger something so for example what you can do is you can set it up so that when the pull request is sent you can tie it out with tools like Jenkin so a tool I think and what it does is it listens for a pull request it's like Oh someone wants you to merge your code to this code base they submit a pull request you can deploy how Jen can deploy that code somewhere in the cloud run the test cases and see if it broke anything so imagine you worked on the

feature you submit a pull request jenkin runs and says oh I your code is broke five features isn't that cool like they automatically found out maybe it worked on your local environment but it broke the staging so it runs it automatically if it passed the test cases then the engineering leave and refused it if it didn't pass then you you're notified and you have to fix your code until the test cases pass right so you can you know go through more in detail but basically you follow that process and this process is called continuous integration

and you know there are more steps but this is kind of a nutshell when people talk about continuous integration that's what they're talking about so that you're working on the code submitting pull requests it runs the test cases automatically so a few times a day even you can get constant feedback like oh okay this is working it's been approved it's automatically integrated and you can have that happen few tons of they continuously your code is being integrated with new features so that's continuous integration it's a cool buzzword but that's all that MIT and all that

means ok continuous delivery means um so once Jenkins tools like jenkin renin the new features didn't break any test cases then you can also set it on so they automatically update staging you're staging and then you know the QA team contested your engineering li can review it you guys can test it and then decide whether you want to update the production server with the latest color so that's a continuous delivery there is still some manual work you can go even step further you have so many test cases but you're so confident that if it met

the test cases and Jenkins said there's no error in any of the test cases then you can go and update the production isn't that amazing that you could just submit a pull request and you can see it live in production all continuously so that's called continuous deployment it's a form of a continuous delivery where you know all the manual process has been removed because your test cases are so strong okay all right let's go into now talk a little bit about managing servers let me pull this out a little bit higher so you can see

this all right let me let's go back to my other screen all right so remember that I can either have bunch of web servers with the operating system and let me let me draw this again so remember I had load balancers bunch of web servers I had on other load balancers and database so it could be if it has the operating system then you're doing it the traditional way all right these are the operating system you need some way to manage this so maybe you have services that automatically manage up and down how you grow

this and you have services to automatically manage this coin up and down right you can you imagine that you can also imagine if I have and all of this was one micro service okay so all of that was a micro service so now that being a micro service so when you think about the micro service just imagine you can have a lot of different computers inside the micro service you may have multiple micro services that need to communicate with each other now right so not only you needed some services that could coordinate some of the

information in between here now you need some things to coordinate information in between so maybe you need another service here so that all the micro service sends data here and then when there is any change it propagates that back to these two that's just another example right so you can imagine that not only do I need services to manage inside my micro service or app I also need to orchestrate how all the different servers have the different services interact with each other alright so I know it is getting a little complex but we're almost to

the end so if you're managing just groups of servers then you can buy the server at Amazon AWS or as your Google found elastic Beanstalk is kind of can be the ones to be the load balancer automatically and scale the server's up and down within that micro service right and there are other services available Microsoft has their own every single major cloud services at their own services that manage that if you're doing containers using Dockers then there are tools available to help you manage those containers so you get the containers and then you you can

there are services that scale up and down the number of containers so those are for example kubernetes and lastly Amazon Elastic container service this is the amazon's version of managing container similar to docker and there are you know farg a community service Azura has one iBM has one third parties like Rancher has this orchestration platform where you can manage you know containers through kubernetes and other things there are other services layer on top for example elasticsearch and you some of these you can think of them as you can have servers use these tools or you

can have micro services talk to each other using these tools but just to give you some example elasticsearch is a search engine based on the Lusine library so maybe you want to record all of the logs of every single server so instead of having one giant server that you create you just use elastic search and have all of the log file be sent to elastic search and then you can do a you can do a search on the elastic search ok Cubana is something that you can add on top of elastic search data to to

visualize some of the log files some of the other things rapid and cue for example is an open source message broker software so instead of once one micro service sending information to another micro service directly you can have some intermediary for example where it can go so just like in this example let's say I have one micro service one micro service another one I have a lot of different microservice I can have some intermediary where the message gets sent over here and then when it's appropriate it sends it to whoever it needs to get sent

to right so you can imagine when we do micro services I have to really modular Rhys and you could create all of those system yourself from scratch or you can see if there is a service which basically means someone else wrote it and you can pay them to use use that to help you manage these containers so that's kind of at a high level how things work it could get I created this in it you know with the goal of providing as much detail note as possible so if you're creating an application or you're going

you're doing interview prep or you're designing your next project you would have some specific tips then you can follow on each step but that caused me to talk a lot longer that I then I usually do but I hope this was helpful so at a high level you know this is the picture you need to work on a project listen before jumping in ask questions research with the limited time put a time cap on how much time you're gonna put into research research some tools and then figure out okay for each micro service that I

need okay what language and framework would be best with the goal that you want to keep the project small with few engineers not have one giant project in one framework with hundreds of developers don't make that mistake modularize it break into micro services for each of the micro service then choose where I'm where am I going to store the data okay using the process that I outline and then for each of the micro service figure out how I'm going to deploy this and how I'm going to use github you know how are you going to

do continuous integration continuous delivery do we need a lot of test cases are we going to use tools like Jenkins and so forth and then once you design each of the micro-service now is time to orchestrate all of the different micro services well just know that there are a lot of tools available or you can build it yourself to help you manage your containers or multiple servers or multiple micro services in a nutshell that's how it works so I hope it was helpful if you found it helpful and if you haven't already you know please

subscribe so you can get alerted of new things but I hope it was helpful and this is what happens in my brain as I think about system architecture so I hope it was helpful for you and I'll put some links that may be helpful for you to go deeper into this top