Ep 025: Fake Results, Real Speed Functional Design In Clojure podcast

1
Ep 118: Pure Parts 28:28

1 year پیش28:28

28:28

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "parts of a pure data model" . We look at pure data models we've created and see what they have in common. Our discussion includes: How we make pure data models How to organize pure data models What are the common parts of a pure data model? What are schemas good for? How is a functional pure data model different than an object-oriented class model? How does a pure data model help with maintenance? Semantic information verses concrete operational information. What about I/O? Input and output transforms for maximizing purity Why save information that the program can't use? Selected quotes And when there's a pure data model, a pure data model is something that is both wide enough to handle an actual use case and be useful, but it's shallow enough that you can understand it and trust the function calls that it has. All of the operations on that data are in the same namespace, so it's easier to understand. I know they're predicates because they end in a question mark. All of the necessary changes and views are encapsulated in a namespace. That means the rest of your application can rely on its higher-level operations when working with the data model. These are a higher-level vocabulary for your application, instead of just Clojure core's vocabulary. Everything that can be done is all co-located in a namespace. Am I multiplying by 0.10 or 0.15? Or am I calculating a tip? One of those statements has more information. A pure data model lets you, as a programmer, think at a higher level in the rest of your application. When you think at a higher level that's trusted, it's a lower cognitive load. You can come back to the code later, read a function, and know what it means in the context of your application. In every pure data model, you have to know what the data looks like. Don't underestimate the value of being able to find places where a predicate is used. It tells you what the code cares about this situation. When you have to nuance the situation, you can look at the call sites and take them all into account. Once you've made the HTTP call, all the information about the request, the response, the body, and all that is pure data. You can do a pure transform from the domain of raw, external HTTP information into the internal domain of the pure data model. But because it's a pure function, it's a lot easier to test. All things are easier to test when they're pure. I/O is a very, very thin layer—both on the way in and the way out. Instead of mixing I/O and logic, do as much I/O as you can, at once, to get a big bag of pure information to work with. And then on the way out, do a pure transform to generate everything you need for the I/O, like the full requests. You can have a big bag of extra context that's there for you as the programmer—even though the program doesn't need it. Parts of a pure model Data tree Schema Literals (eg. initial state) Predicates Data operations Reducing function (state + event) Transforms in Transforms out Views (special kind of transform) Links "Reentrant Coding" Series Ep 114: Brand New, Again Ep 115: The Main Event Ep 116: The Main Focus Ep 117: Pure Understanding…

1
Ep 117: Pure Understanding 25:13

1 year پیش25:13

25:13

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "pure data models" . We find a clear and pure heart in our application, unclouded by side effects. Our discussion includes: What is the heart of a Clojure application? Pure data models! What is a pure data model? Why do we use pure data models? How do they compare to object-oriented data models? Where do you put pure data models? How do you organize your code? How pure data models avoid object-oriented dependency hell. How do pure data models help you understand the codebase quickly? Why does a codebase become easier to reason about by using pure models? How do pure models fit into the overall application? How do pure models relate to state and I/O? Examples of pure models Selected quotes It's functional programming, so we're talking about pure data models! That is our core, core, core business logic. A pure data model is pure data and its pure functions. No side effects! We already have a whole set of Clojure core functions to operate on data, so why would we have functions that are associated with just this pure data? Because you want to name the operations, the predicates, and all the other things to do with this data, so that you, as a human, understand. Those functions are high-level vocabulary that you can use to think about your core model. They are business-level functions. They are super-important, serious functions. We don't like side effects, so we define an immutable data structure and functions that operate on that data. They cannot update that data. They can't change things in place. They always have to return a new version of it. At a basic level, you have functions that take the data. They give you a new data tree or find something and return it. We like having the app.model namespace. You can just go into the app/model folder and see all of the core models for the whole application. Any part of the application can have access to the model. The functions are the interface. All you can do is call functions with pure data and get pure data back. You can't mess anything up except your own copy. It's just a big pool of files that are each a cohesive data model. They're a resource to the whole application, so anything that needs to work with that data model will require it and have all the functions to work with it. With pure models, there's no surprise! In OO, the larger these object trees get, the more risk there is. Any new piece of code, in the entire codebase, has access to the giant tree of objects and can mess it up for everything else. Pure models lower your cognitive load. The lower the load is, the more your brain can focus on the actual problem. You can read the code and take it at face value because the function is 100% deterministic given its inputs. If it's a pure function, you don't have to wonder what else is happening. The model directory is an inventory of the most important things in the entire application. Here are all the things that matter. As much code as possible should be in pure models. Look at the unit tests for each pure model to understand how the application reasons and represents things. It's the very essence of the application. A lot of times in functional communities, we say "keep I/O at the edges." Imagine one of these components is like a bowl. At the first edge, there's I/O. In the middle is the pure model goodness. On the other side is I/O again. None of the I/O is hidden. That's the best part. Because I/O isn't hidden behind a function, it's easier to understand. Cognitive load is lower. You can read the code and understand it when you get back into it and you're fixing a bug. The shallower your I/O call stacks are, the easier they are to understand. Where there are side effects, you want very, very shallow call stacks, and where there are no side effects, and you can unit test very thoroughly, you don't have to worry about the call stack as much. Links "Reentrant Coding" Series Ep 114: Brand New, Again Ep 115: The Main Event Ep 116: The Main Focus Ep 029: Problem Unknown: Log Lines…

1
Ep 116: The Main Focus 22:44

1 year پیش22:44

22:44

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "frontend matters" . We turn our attention to the frontend, and our eyes burn from the complexity. Our discussion includes: What is a Single Page Application (SPA)? The progression from static websites, to multi-page websites, to JavaScript-enhanced pages, to SPAs. Why we like SPAs. Where's the "main" for the frontend? The complexity of frontend builds. How does the frontend app start running? How does the "table of contents" concept apply on the frontend? How can you make the most important parts of the application clear? What are the most important parts of a SPA? What is the "backend of the frontend"? The asynchronous, reactive cycle that drives a SPA. Selected quotes You don't usually go into a code base just to browse it, or just to have fun. You go there with a purpose. You need to work. You need to get something done fast. We like rich frontends. We're able to do a lot more interactivity. There's less interruption when the page has to load. There are a lot of advantages to SPAs. With a SPA, it's really, really fast to switch between everything. It feels almost instantaneous because there is almost nothing to load each time. The counterpart is that a SPA is more sophisticated, so it ends up being more complicated. It's almost like a process that's running continuously. There's more code that's present in a SPA than any individual page load. From the browsers point of view, the "main" is the markup, and you have to tell it to run some code. It's just one blob of code to the browser. You can't look at that code because it's transpiled, minified JavaScript. I do think it's interesting that we've gotten several minutes into this episode, and we're still talking about how things get made into the final sausage. It's reflective of how much effort it takes to set up the JavaScript ecosystem. We make a "main.cljs" file, and that is the top of the application. It's a signpost. "Hey! Hey! Look here first!" The tab's not going to go away, so all we need to do is start up all the event listeners because JavaScript is a very event-driven language. I want "main" to be a table of contents of everything that matters in the app: the views, the routes, the URLs, browser hooks, web sockets, etc. The worst kind of "main" is no "main" at all. There are frameworks where you make a whole bunch of separate files for each of your routes. I love how many times we said the word "react" in this episode. It's all very event driven. That's just the model of the whole browser. It's the water that you swim in, so you must swim the right way in order for the application to succeed. Reactive cycle User Interaction → Event → Callback → Reactive Model → Re-Render Kinds of frontend components Navigation (eg. browser history) Router Settings State holders (eg. app state, reactive atoms) Pure models Browser integrations (eg. camera, microphone, GPS, notification API, fullscreen API) Render Links "Reentrant Coding" Series Ep 114: Brand New, Again Ep 115: The Main Event Clojure for static sites Stasis Powerpack ClojureScript for frontends shadow-cljs Figwheel Reagent…

1
Ep 115: The Main Event 28:42

1 year پیش28:42

28:42

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "the main function" . We look for a suitable place to dig into the code and find an entry point. Our discussion includes: We're diving into a Clojure project. Where should we start? First up: the backend. Why start with the code? Why is "main" so important? What's the problem with lots of entry points? What should a good "main" do? How does a good "main" help you find things? What is a good component system? The difficulty of understanding inter-component dependencies. What is a "table of contents" for your program? Random access reading. What are common kinds of components? What is a pure model? How do pure parts and side-effecting parts all come together? Selected quotes Be friendly to people that come into your code base. I didn't trust myself as much as I should. "You just start at the top and write from the top to the bottom." "That's how I code everything. It's just one really large file." So all parts of your code are reachable from "main"—or should be. A great "main" is where you can see all the major parts of the application and how they fit together with each other. A terrible "main" is a system that doesn't have any "main" at all! It has a thousand different entry points that are all over the place. A great "main" is very compact. You can scan it. It's a very high level recipe of what's going on. Component has a system map. You can just look at the data structure and see all of the different components—the major players. The alternative is components that declare dependencies on each other. It's a kind of nightmare. Everything running independently, calling or referencing each other. What's using what? What's calling what? How does information flow through? When dependency information gets spread all over the place, you have to go to all the different places to even understand what you need. Having it all in one place is essential for understanding. It really helps when you can see the interdependency between things really easily. Each component should only get what it actually needs. It shouldn't just get the whole map of every dependency. Common kinds of components: shared resources, integrations in, integrations out, pure models, state holders, and amalgamations. It all comes together in the amalgamations. Pure models are the core of the application. They are higher level than just data manipulation. All of the actual nuts and bolts is in the pure models, and that makes the components relatively light. The goal is to make the pure model as fat as possible without introducing any non-determinism, AKA side effects. Amalgamations: it's where the "in", the "out", and the pure model all come together—where the real work gets done. The amalgamation components end up being at the middle of the application. They're a kind of orchestrator. Common kinds of components shared resources integrations in integrations out state holders pure models amalgamations Links Component Ep 097: Application of Composition . We discuss having table-of-contents style sections in your code. It helps with finding things and forming the big picture.…

1
Ep 114: Brand New, Again 16:08

1 year پیش16:08

16:08

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "what's old is new again" . We find ourselves staring at code for the first time—even though we wrote some of it! Our discussion includes: Christoph re-opened his own code base after 14 months! Nate joined a software team with a large, legacy code base. Why does the problem of "fresh eyes" matter so much? How is my past self going to treat me? Are the past teammates going to help me? Why you shouldn't rely on memory. What's involved in really understanding a code base in order to make a change? The frustration of being unproductive. How is an app put together? What's the flow of information? How do you really know if your code is comprehensible? Selected quotes It's always fun to start something new. You don't have to worry about all those other things from the past! I was a little nervous about that other guy who made all the code: me fourteen months ago! I wasn't sure how good of a teammate he was going to be. The word "legacy" is like a big bad word, but I feel like it should be a badge of honor! It's software that is running right now and paying your salary! You should have a reverence for code that exists and is running. At some point in time, you or a new team member, is diving into a code base with fresh eyes. It brings up so many important issues that we face as developers. We spend so much time reading code and forming mental models about what is going on. A fundamental challenge in software development is understanding, comprehending and reasoning about the code base. Comprehending and reasoning about the code is one of the primary drivers behind the "why" of a lot of the so-called "best practices" of the industry. Why do you write tests? Why do you write documentation? Why do you try to have a good design in your application? There's this constant learning that we have to do, and so try to make that easier. He moved on to better projects in the sky. We've lost him to a better project in the Cloud. He moved to a better project upstate! It's easy to say: "We have great documentation! Our code is super readable! Decoupled? Absolutely! Our pure data models? Totally comprehensible!" It's easy to say that, but you really find out if those things are true when somebody new joins the team or when you have to revisit the code after a long time. Always trying to teach someone else about your code. There's always some future person. What can we do now as we're setting up the situation for new people in the future?…

1
Ep 113: Highlightify! 27:13

1 year پیش27:13

27:13

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "highlight highlights" . We highlight the highlights of the Sportify! series. Our discussion includes: Sportify: The Highlights! Why is Clojure so well suited for situated problems? Why should you work with real-world data ASAP? How does REPL-driven development move seamlessly from exploration to implementation? Don't invent your domain model, let it emerge! How do you grow the application domain? How does Clojure help facilitate bottom-up development? Why is early integration just as important as exploration? How is reliability related to frequency of use? How does pureness affect reliability? What is the think-do-assimilate pattern? Are pure functions even needed in I/O heavy applications? How do you shift a code base from imperative to pure? Why is data-centric simpler than behavior-centric? Selected quotes "Let's put it all together into one context to rule them all and in the darkness bind them!" "But we're trying to spread the light of sports highlights across the Internet!" There's nothing like actually seeing real data come from real APIs. No amount of talking to your boss or talking to the intern or reading documentation can replace what you get from touching the real-world situation. Clojure helps you figure out how to bring the pieces together because you can just run the pieces in an ad-hoc way. You can just work on each of the parts without having to unify them into some kind of global proof system that's being foisted on you by static analysis. It's like whiplash-driven development: you're moving so fast, you have to take a break just to take a breath! The bottom-up way of constructing in Clojure has two properties: you're grounded in the real world, and you're just making what you need as you go along. It's very efficient. Some of that code is going to find its way into your final working solution. You're always making progress. You're always grounded in reality. You're just building what you need as you go along. It's not wasteful. It's very iterative. Very lean. Always forward motion. If your system exploration is in Clojure, you can cross information streams a lot easier than if you're using separate tools. In Clojure, it's all data. You can just hand data back and forth. You're not only discovering the properties of each information silo you're working with, but you're discovering the properties of how that data might fit and merge together. It's a grounded, incremental process for each of the parts, but also as the parts come together. Sometimes you don't know what the final solution is going to be even though you have all the necessary parts. It's greater than the sum of its parts. It isn't until you start running things over and over more frequently that you begin to discover the smaller percentage reliability issues. The way you increase reliability is by minimizing things with side effects and maximizing things that are pure. You're learning, at every step, what point needs more reliability. The more pure data you have, the more visibility you have. The more pure functions you have, the more testability you have. So, reliability and pureness are definitely related. It is amazing how much opportunity there is to move things into pure functions. The actual fetching or querying of the thing ends up constituting a pretty small part of your application. Working with the data tends to dominate. The think-do-assimilate pattern allows you to maximize the testable surface in your application by factoring all the I/O out. You start minimizing the I/O parts because your domain has emerged. Links Sportify! series: Ep 101: Sportify! Ep 102: REPLify! Ep 103: Explorify! Ep 104: Assembleify! Ep 105: Codify! Ep 106: Robustify! Ep 107: Idempotify! Ep 108: Testify! Ep 109: Extractify! Ep 110: Contextify! Ep 111: Loopify! Ep 112: Purify!…

1
Ep 112: Purify! 33:20

1 year پیش33:20

33:20

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "pure data, pure simplicity" . We loop back to our new approach and find more, and less, than we expected! Our discussion includes: Sportify reaches the end zone! How can you make software extremely reliable? How can you keep logic pure when it depends on the outcome of I/O? What is the "think, do, assimilate" pattern? What is a "context"? What information should it contain? What is an "operation"? What information should it contain? Why does "structural sharing" matter? When to use multimethods vs case statements. The benefits of reducing functions. Testing with mocks vs data. Why test with real-world data? How does minimizing I/O make your code testable, inspectable, and repeatable? How to support any process conditional. A general way to model any process. Selected quotes Let's get some Hammock Time. We're big fans of Hammock Time! We make a function for each of those: think, do, assimilate. A function to figure out the next thing, a function to do it, and a function to integrate the results back into the context. ("determine-operation", "execute-operation", and "update-context".) It's not a single point of failure! It's a single point of context. Where you have a binding in a let block, you have a key in a context map. There's a symmetry there. You can make the operation map as big, fat, and thick as you want, so "execute-operation" has 100% of everything it needs for that operation. The "determine-operation" function can decide anything because it has the full context—the full world at its disposal! Clojure has structural sharing, so all the information is cheap. We can keep a bunch of references to it in memory. We're not going to run out of memory if we keep information about every step. The "update-context" is a reducer, so we can make a series of fake results in our test and run through different scenarios using "determine-operation" and "update-context". We're able to test all of our logic in our test cases because we can just pass in different types of data. Your tests are grounded in reality. They're grounded in what has happened. We've aggressively minimized the side effects down to the tiniest thing possible! Data is inert. Mocks are not. Mocks are behavior. You can just literally copy from the exception and put it in your test. There's no need transform it. It is already useful. It's very testable. It's very inspectable. It's very repeatable. It creates a really simple overall loop. You want those I/O implementations so small and dumb that the only way to mess them up is if you're calling the wrong function or you're passing the wrong args. Once it works, it will always work, and you no longer have to test it. We need to build into context every little bit of information we need to know to make a decision. Context takes anything that is implicit and makes it 100% explicit, because you can't get data across the boundaries without putting it in the context. You have no option but to put everything in the context, so you know everything that's going on. We're in this machine, and there's no exit. We're on the freeway, and there's no off-ramp. We're in the infinite loop! How do we know we're done? How do we know we're done? Links Sportify! series: Ep 101: Sportify! Ep 102: REPLify! Ep 103: Explorify! Ep 104: Assembleify! Ep 105: Codify! Ep 106: Robustify! Ep 107: Idempotify! Ep 108: Testify! Ep 109: Extractify! Ep 110: Contextify! Ep 111: Loopify! "Hammock Driven Development" by Rich Hickey core.async Portal…

1
Ep 111: Loopify! 28:54

1 year پیش28:54

28:54

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "trying again" . We throw our code in a loop, and it throws us for a loop. Our discussion includes: Sportify continues! When is it time to stop developing? How do we handle retries? What if you need to recur from catch ? How do we recover mid-process? Where should the recovery logic go? Is there a way to get all the critical context at the same level? What should you preserve across a recur ? What does it mean to be "loop native"? What is the basic structure for any automation? What is a "single application state"? Selected quotes It's a lot like having a project on a workbench. You have all of the tools and all the information laid out before you on that workbench. Nothing is tucked in a drawer or inside a cabinet. That's a very important lesson for any developer: you can always stop—at least after it's working. Nothing in the world is solved except by adding another level of abstraction. I was not expecting that level of mutation! I was expecting a Kafka log written in stone! The positive is it has everything. The negative is it has everything. We would like more loop-native code inside of our cloud-native application. Are you suggesting that just because we can, it doesn't mean we should? We're programmers! If the language lets us do it, it must be a good idea! One of the reasons why I like Clojure is because it specifically tells me that I can't do some things that are bad to do. All of the context is in one map. It has everything in it. One map to rule them all! Might this be the fabled "single application state"? We have the thinking function, the doing function, and the assimilate function. Links Sportify! series: Ep 101: Sportify! Ep 102: REPLify! Ep 103: Explorify! Ep 104: Assembleify! Ep 105: Codify! Ep 106: Robustify! Ep 107: Idempotify! Ep 108: Testify! Ep 109: Extractify! Ep 110: Contextify!…

1
Ep 110: Contextify! 23:22

1 year پیش23:22

23:22

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "gathering debugging context" . Our downloads fail at random, but our dead program won't give us any answers. Our discussion includes: Sportify! Now, with even more exceptions! How do you know the state of the application after it has already died? Where do you catch the exception? What do you log and where? How do you get the context to where you need it? What place has all the important context? Where does the pure logic belong? How can you keep I/O functions ruthlessly simple? The beauty of fully described operations. Selected quotes We just need to fill out our support ticket and say, "Hey! Fix your service!" It couldn't possibly be our code! So there is an error happening, but what happened just before that error? It is dead. There's no way to ask it any questions. It will not give us any answers. The only way to know what the program was doing, is to know what the program was doing. If you're trying to figure out what the program was doing by reverse engineeringing it, you're going to get it wrong. I love hiding side effects with macros! That's one of my favorite things to do in Clojure! It makes me feel like I'm using Scala again! We don't want the I/O function to do any thinking of any kind. It's a grunt. We fully specify the bits it needs to know. It's 100% a boring outcome of what we passed into it. Those I/O functions end up being ruthlessly simple. They're often just one line! We remove the thinking, so we remove the information. It's not because we don't like pure functions. We put them in a place where we can have all the information in one place. We're getting to the point where our let block is getting really long really—maybe too long. We're really letting ourself go! Links Sportify! series: Ep 101: Sportify! Ep 102: REPLify! Ep 103: Explorify! Ep 104: Assembleify! Ep 105: Codify! Ep 106: Robustify! Ep 107: Idempotify! Ep 108: Testify! Ep 109: Extractify! Cognitect AWS API…

1
Ep 109: Extractify! 26:35

1 year پیش26:35

26:35

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "separating data from I/O" . We need to test our logic, but the I/O is getting in the way. Our discussion includes: Bugs in our Clojure code? What?! You must mean Javascript. When does Hammock Time not help? How logic grows and expands. Why you should test your logic. What parts do not need testing? What's a functional approch to working with APIs? How do you separate out logic for complicated I/O sequences? I/O testing without mocks. Why do we create our own model when Java gives us a class model already? The problems of built ins. Selected quotes We're using Clojure. Everything should be perfect, right?! I love Hammock Time for figuring out hard problems, but in this case, I think we have a simple problem of testing. You got to have the right amount of celebration after all those "line crossings" and "goal scorings" and stuff. We're doing a relatively simple process: we're downloading things and compiling them together into a file. But, it's amazing just how much logic is all throughout this process. As soon as you make a process, there's always going to be people who want to do it differently! If experience is any indicator, you always need more information. One of the reasons why you test is, when you make this kind of logic change, you want to make sure that everything continues to function. You need to write tests so that when you make future changes, your old self is there sitting right next to you making sure that the old use cases are all covered, so that you only have to think about the new use cases. With REPLing, you're figuring it out. With tests, you're locking it down and making sure that you have coverage in different situations. Our biggest obstacle here is that logic and I/O are mixed up together. Wait! Wait! We want to test our code. We don't want to spend our life writing code. Did you write the mock correctly? How do you write a test for the mock? I think we need to completely pivot our approach here. The problem is that we have I/O, logic, I/O, logic, I/O, logic. We have those two things right next to each other. What we should do instead is completely invert our thinking. Let's gather information and then we can do pure logic on that data. Separate those two things. We're going to extract from those POJOs. [Groan] I've got to use these terms every now and again or else I'm going to forget them all. So we do an I/O call, collect information, and create our own internal representation. We just need a few bits of it, so we create a working representation of that. It's our representation. It's our program's way of looking at the world. Craft the different scenarios in data that represent all the real life situations we found. One of the problems of using built-ins is: what parts matter? We're accreting working information into a larger and larger context. You're setting the table with all the pieces that are defined in your working world and then creating unit tests in terms of those. The world was like, "Hold my beer!" Links Sportify! series: Ep 101: Sportify! Ep 102: REPLify! Ep 103: Explorify! Ep 104: Assembleify! Ep 105: Codify! Ep 106: Robustify! Ep 107: Idempotify! Ep 108: Testify!…

1
Ep 108: Testify! 22:42

1 year پیش22:42

22:42

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "testing around I/O" . We start testing our code only to discover we need the whole world running first! Our discussion includes: How do you unit test an I/O heavy process? Should you be REPL-driven or test-driven? What is the REPL suited for? What are tests suited for? What do you need to know to figure out the bug? How can a purely functional language help with testing? Techniques for factoring out pure logic. What is an extraction function? What is an ingestion transform? Outside data models verses "internal" or "working" models. Code smells when working with external data. Where can you use schemas in your code? Selected quotes The tracer bullet misfires every now and again. Now you're going from a tracer bullet to a silver bullet—apparently trying to solve all the problems at once! The REPL lets you figure out the basics of the process and your own way of thinking about it and modeling it, and the tests let you start handling more and more cases. Exploration early, testing later. Are you just supposed to log everything all the time? Always run your code with a profiler attached? If you look between each I/O step, there is pure connective tissue that holds those things together. We remove the logic and leave just the I/O by itself. With pure functions, we don't have to worry about provisioning the AWS cluster for the tests to run! It's really tempting to use the external data as your working data. What is the data that this application reasons on? By creating an extractor function, you pull all of the parts that matter into a single place. It returns a map for that entity that you can reason on and schema check. We've distilled out the sea of information into a drinkable cupful. We've gone from the mountain spring to bottled water. I guess you could always take all the raw data and shove them off in an Elasticsearch instance for massive debugging later—in some super-sophisticated implementation. Not how do we accomplish it, but how do we test it? Links Sportify! series: Ep 101: Sportify! Ep 102: REPLify! Ep 103: Explorify! Ep 104: Assembleify! Ep 105: Codify! Ep 106: Robustify! Ep 107: Idempotify! Related episodes: Ep 027: Collected Context Ep 029: Problem Unknown: Log Lines…

1
Ep 107: Idempotify! 31:53

1 year پیش31:53

31:53

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "handling endless errors" . We discover when giving up is the way to get ahead. Our discussion includes: Finishing up our Sportify tracer-bullet implementation. What context can you put in a file name? Tricks on working with time information. Trying to "fix" a situation vs adding more context. What is a "positive representation" of a problem? What are "fundamental" versus "derived" facts? What information tends to be more stable? What information should you try to co-locate? How do you handle resources that are not currently available? How many times should you retry? How long should you wait between retries? How do you figure that out? What is idempotency? How can it help? At what point should you stop trying to handle errors? Can you have too much automation? How can you troubleshoot intermittent problems? Selected quotes We don't need a time zone offset because we know it's in UTC! In the spirit of building up the language to meet our domain, we can write a pure function! We want a deterministic way to go from this kind of common information into all the other bits of derived information. I was really hoping that we would finally be done with the errors and we could just get a highlight clip, but yet again, the world has conspired against us to make our life difficult as a programmer! Hold on! Hold on! The first thing we should do is run our process again because maybe the error will just go away! The problem does not go away in this instance. The problem just goes back to being hidden! The frustrating thing about programming is that code will do exactly what you told it to do. Clojure is already positive about nothing, that's what we call nil , so why not be positive about bad stuff too? Now I'm personifying the function as myself! It's better to reference information by its main identifier as opposed to some derived identifier. The context is all over the place: some context in memory, some context in the file system, and some intermediate context in the imperative function. That's the last problem we encounter, right? You never know what the world's going to throw at you! Why don't we just run it again? Let's just run it again! Maybe it'll be ready now... Maybe now... Maybe it will be ready now... And then we hit another error which is: the MAM rejects us for making too many requests! How long do you wait? Should you back off? That adds a lot of complexity to the code at that level. Because adding idempotency increases the complexity of that part of the solution, we only want to add it where it's necessary. The error condition is happening right now, so let's write the code to fix it right now. That's the way automation is at some point in time: a human needs to do it. You can find yourself in a situation where you're trying to do too many automatic things. Is it really worth doing? You cannot solve every possible permutation once you're interacting with the real world. It's okay to put up the guardrails, and if I'm outside them, robot me throws up my hands and says, "Human please!" But what happens when you run it again and the clip is ready? Now your retry logic doesn't get tested. Links Sportify! series: Ep 101: Sportify! Ep 102: REPLify! Ep 103: Explorify! Ep 104: Assembleify! Ep 105: Codify! Ep 106: Robustify! Related episodes: Ep 027: Collected Context Ep 029: Problem Unknown: Log Lines…

1
Ep 106: Robustify! 30:37

۲ years پیش30:37

30:37

۶ years پیش35:06

35:06

Nate is dropped in the middle of a huge log file and hunts for the source of the errors. We dig into the world of DonutGram. "We are the devops." Problem: we start seeing errors in the log. "The kind of error that makes total sense to whoever wrote the log line." (04:10) We want to use Clojure to characterize the errors. Why not grep? Well...we grep out the lines, count them, and we get 10,351 times. Is that a lot? Hard to say. We need more info. (05:50) Developers think it's safe to ignore, but it's bugging us. We want to come up with an incident count by user. "Important thing we do a lot in the devops area: we see data and then we have to figure out, what does it mean?" "What story is the data telling us?" The "forensic data" versus "operating data" (Ep 022, 027) "With forensic data, you're trying to save the state of the imprint of the known universe over time." (07:40) The log file is the most common forensic data for applications. To characterize, we need to manipulate the data at a higher level than grep. First goal: make the data useful in Clojure Get a nice sample from the production log, put it in a file, load up the REPL, and start writing code to parse it. Need to parse out the data from the log file and turn it into something more structured. (12:05) Let's make a function, parse-line that separates out all the common elements for each line: timestamp log level thread name package name freeform "message" "A message has arrived from the developers, lovingly bundled in a long log line." (13:10) We can parse every line generically, but we need to parse the message and make sense of that. "We want to lift up the unstructured data into structured data that we can map, filter, and reduce on." We will have different kinds of log lines, so we need to detect them and parse appropriately. We want to amplify the map to include the details for the specific kind of log line. We'll use a kind field to identify which kind of log line it is. There are two steps: recognize which kind of log line the "message" is for parse the data out of that message "Maybe we'll have a podcast that's constantly about constantly . It's not just juxt ." (18:45) How do we do this concisely? Let's use cond : flatten out all the cases code to detect kind (the condition) code to parse the details (the expression) Can use includes? in the condition to detect the kind and re-matches to parse. (20:00) Why not use the regex to detect the kind too? Can avoid writing the regex twice by using a def . It's less code, but now we're running the regex twice: once in the condition and once to parse. We can't capture the result of the test in cond . No "cond-let" macro. We could write our own cond-let macro, but should we? "When you feel like you should write a macro, you should step back and assess your current state of being. Am I tired? Did I have a bad day? Do I really need this?" (24:05) New goals for our problem: one regex literal only run the regex once keep the code that uses the matches close to the regex Similar problem to "routes" for web endpoints: want the route definition next to the code that executes using the data for that route. Feels like an index or table of options. (25:20) Let's make a "table". A vector of regex and handler-code pairs. We need a coffee mug: "Clojure. It's just a list." The code can be a function literal that takes the matches and returns a map of parsed data. Write a function parse-details which goes through each regex until one matches and then invokes the handler for that one. (See below.) (30:15) Once we have higher-level data, it's straight forward to filter and group-by to get our user count. Once again, the goal is to take unstructured data and turn it into structured data. "You have just up-leveled unstructured information into a sequence of structured information." Can slurp , split , map , filter , and then aggregate. (32:10) What happens when we try to open a 10 GB log file? Sounds like a problem for next week. "Fixing production is always a problem for right now." "If a server falls over and no one outside of the devops team knows about it, did the server really fall over?" "Who watches the watchers?" Related episodes: 020: Data Dessert 022: Evidence of Attempted Posting 027: Collected Context Clojure in this episode: #"" re-matches case cond if-let ->> slurp filter group-by def constantly juxt clojure.string/ includes? split Related links: GraalVM Code sample from this episode: ( ns devops.week-01 ( :require [ clojure.java.io :as io ] [ clojure.string :as string ] )) ;; General parsing ( def general-re # "(\d\d\d\d-\d\d-\d\d)\s+(\d\d:\d\d:\d\d)\s+\|\s+(\S+)\s+\|\s+(\S+)\s+\|\s+(\S+)\s+\|\s(.*)" ) ( defn parse-line [ line ] ( when-let [[ whole dt tm thread-name level ns message ] ( re-matches general-re line )] { :raw/line whole :log/date dt :log/time tm :log/thread thread-name :log/level level :log/namespace ns :log/message message })) ( defn general-parse [ lines ] ( ->> lines ( map parse-line ) ( filter some? ))) ;; Detailed parsing ( def detail-specs [[# "transaction failed while updating user ([^:]+): code 357" ( fn [[ _whole user ]] { :kind :code-357 :code-357/user user })] ]) ( defn try-detail-spec [ message [ re fn ]] ( when-some [ matches ( re-matches re message )] ( fn matches ))) ( defn parse-details [ entry ] ( let [{ :keys [ log/message ]} entry ] ( if-some [ extra ( ->> detail-specs ( map ( partial try-detail-spec message )) ( filter some? ) ( first ))] ( merge entry extra ) entry ))) ;; Log analysis ( defn lines [ filename ] ( ->> ( slurp filename ) ( string/split-lines ))) ( defn summarize [ filename calc ] ( ->> ( lines filename ) ( general-parse ) ( map parse-details ) ( calc ))) ;; data summarizing ( defn code-357-by-user [ entries ] ( ->> entries ( filter #( = :code-357 ( :kind % ))) ( map :code-357/user ) ( frequencies ))) ( comment ( summarize "sample.log" code-357-by-user ) ) Log file sample: 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | transaction failed while updating user joe: code 357 2019-05-14 16:48:56 | process-Poster | INFO | com.donutgram.poster | transaction failed while updating user sally: code 357 2019-05-14 16:48:57 | process-Poster | INFO | com.donutgram.poster | transaction failed while updating user joe: code 357…

94 subscribers

مشابه Functional Design in Clojure

Peacock TV

Minecraft

BIODANCE Bio-Collagen Real Deep Mask, Hydrating Overnight Hydrogel Mask, Pore Minimizing, Elasticity Improvement, 34g x4ea

پادکست هایی که ارزش شنیدن دارند

Functional Design in Clojure « » Ep 025: Fake Results, Real Speed

Ep 025: Fake Results, Real Speed

پادکست هایی که ارزش شنیدن دارند

به Player FM خوش آمدید!

Mini Mic Pro Wireless Microphone for iPhone, iPad, Android, Lavalier Microphone for Video Recording - 2 Pack iPhone Mic Crystal Clear Recording with USB-C for Podcast, ASMR

LifeStraw Personal Water Purifier for Hiking, Camping, Travel, and Emergency Preparedness

PicassoTiles 100 Piece Set 100pcs Magnet Building Tiles Clear Magnetic 3D Blocks Construction Playboards, Creativity Beyond Imagination, Inspirational, Recreational, Educational Conventional

$25 PlayStation Store Gift Card [Digital Code]

مشابه Functional Design in Clojure

Functional Design in Clojure « »
Ep 025: Fake Results, Real Speed