Ep 078: Impartial Thoughts Functional Design In Clojure podcast

F

Functional Design in Clojure

1
Ep 118: Pure Parts 28:28

1 year پیش28:28

28:28

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "parts of a pure data model" . We look at pure data models we've created and see what they have in common. Our discussion includes: How we make pure data models How to organize pure data models What are the common parts of a pure data model? What are schemas good for? How is a functional pure data model different than an object-oriented class model? How does a pure data model help with maintenance? Semantic information verses concrete operational information. What about I/O? Input and output transforms for maximizing purity Why save information that the program can't use? Selected quotes And when there's a pure data model, a pure data model is something that is both wide enough to handle an actual use case and be useful, but it's shallow enough that you can understand it and trust the function calls that it has. All of the operations on that data are in the same namespace, so it's easier to understand. I know they're predicates because they end in a question mark. All of the necessary changes and views are encapsulated in a namespace. That means the rest of your application can rely on its higher-level operations when working with the data model. These are a higher-level vocabulary for your application, instead of just Clojure core's vocabulary. Everything that can be done is all co-located in a namespace. Am I multiplying by 0.10 or 0.15? Or am I calculating a tip? One of those statements has more information. A pure data model lets you, as a programmer, think at a higher level in the rest of your application. When you think at a higher level that's trusted, it's a lower cognitive load. You can come back to the code later, read a function, and know what it means in the context of your application. In every pure data model, you have to know what the data looks like. Don't underestimate the value of being able to find places where a predicate is used. It tells you what the code cares about this situation. When you have to nuance the situation, you can look at the call sites and take them all into account. Once you've made the HTTP call, all the information about the request, the response, the body, and all that is pure data. You can do a pure transform from the domain of raw, external HTTP information into the internal domain of the pure data model. But because it's a pure function, it's a lot easier to test. All things are easier to test when they're pure. I/O is a very, very thin layer—both on the way in and the way out. Instead of mixing I/O and logic, do as much I/O as you can, at once, to get a big bag of pure information to work with. And then on the way out, do a pure transform to generate everything you need for the I/O, like the full requests. You can have a big bag of extra context that's there for you as the programmer—even though the program doesn't need it. Parts of a pure model Data tree Schema Literals (eg. initial state) Predicates Data operations Reducing function (state + event) Transforms in Transforms out Views (special kind of transform) Links "Reentrant Coding" Series Ep 114: Brand New, Again Ep 115: The Main Event Ep 116: The Main Focus Ep 117: Pure Understanding…

F

Functional Design in Clojure

1
Ep 117: Pure Understanding 25:13

1 year پیش25:13

25:13

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "pure data models" . We find a clear and pure heart in our application, unclouded by side effects. Our discussion includes: What is the heart of a Clojure application? Pure data models! What is a pure data model? Why do we use pure data models? How do they compare to object-oriented data models? Where do you put pure data models? How do you organize your code? How pure data models avoid object-oriented dependency hell. How do pure data models help you understand the codebase quickly? Why does a codebase become easier to reason about by using pure models? How do pure models fit into the overall application? How do pure models relate to state and I/O? Examples of pure models Selected quotes It's functional programming, so we're talking about pure data models! That is our core, core, core business logic. A pure data model is pure data and its pure functions. No side effects! We already have a whole set of Clojure core functions to operate on data, so why would we have functions that are associated with just this pure data? Because you want to name the operations, the predicates, and all the other things to do with this data, so that you, as a human, understand. Those functions are high-level vocabulary that you can use to think about your core model. They are business-level functions. They are super-important, serious functions. We don't like side effects, so we define an immutable data structure and functions that operate on that data. They cannot update that data. They can't change things in place. They always have to return a new version of it. At a basic level, you have functions that take the data. They give you a new data tree or find something and return it. We like having the app.model namespace. You can just go into the app/model folder and see all of the core models for the whole application. Any part of the application can have access to the model. The functions are the interface. All you can do is call functions with pure data and get pure data back. You can't mess anything up except your own copy. It's just a big pool of files that are each a cohesive data model. They're a resource to the whole application, so anything that needs to work with that data model will require it and have all the functions to work with it. With pure models, there's no surprise! In OO, the larger these object trees get, the more risk there is. Any new piece of code, in the entire codebase, has access to the giant tree of objects and can mess it up for everything else. Pure models lower your cognitive load. The lower the load is, the more your brain can focus on the actual problem. You can read the code and take it at face value because the function is 100% deterministic given its inputs. If it's a pure function, you don't have to wonder what else is happening. The model directory is an inventory of the most important things in the entire application. Here are all the things that matter. As much code as possible should be in pure models. Look at the unit tests for each pure model to understand how the application reasons and represents things. It's the very essence of the application. A lot of times in functional communities, we say "keep I/O at the edges." Imagine one of these components is like a bowl. At the first edge, there's I/O. In the middle is the pure model goodness. On the other side is I/O again. None of the I/O is hidden. That's the best part. Because I/O isn't hidden behind a function, it's easier to understand. Cognitive load is lower. You can read the code and understand it when you get back into it and you're fixing a bug. The shallower your I/O call stacks are, the easier they are to understand. Where there are side effects, you want very, very shallow call stacks, and where there are no side effects, and you can unit test very thoroughly, you don't have to worry about the call stack as much. Links "Reentrant Coding" Series Ep 114: Brand New, Again Ep 115: The Main Event Ep 116: The Main Focus Ep 029: Problem Unknown: Log Lines…

F

Functional Design in Clojure

1
Ep 116: The Main Focus 22:44

1 year پیش22:44

22:44

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "frontend matters" . We turn our attention to the frontend, and our eyes burn from the complexity. Our discussion includes: What is a Single Page Application (SPA)? The progression from static websites, to multi-page websites, to JavaScript-enhanced pages, to SPAs. Why we like SPAs. Where's the "main" for the frontend? The complexity of frontend builds. How does the frontend app start running? How does the "table of contents" concept apply on the frontend? How can you make the most important parts of the application clear? What are the most important parts of a SPA? What is the "backend of the frontend"? The asynchronous, reactive cycle that drives a SPA. Selected quotes You don't usually go into a code base just to browse it, or just to have fun. You go there with a purpose. You need to work. You need to get something done fast. We like rich frontends. We're able to do a lot more interactivity. There's less interruption when the page has to load. There are a lot of advantages to SPAs. With a SPA, it's really, really fast to switch between everything. It feels almost instantaneous because there is almost nothing to load each time. The counterpart is that a SPA is more sophisticated, so it ends up being more complicated. It's almost like a process that's running continuously. There's more code that's present in a SPA than any individual page load. From the browsers point of view, the "main" is the markup, and you have to tell it to run some code. It's just one blob of code to the browser. You can't look at that code because it's transpiled, minified JavaScript. I do think it's interesting that we've gotten several minutes into this episode, and we're still talking about how things get made into the final sausage. It's reflective of how much effort it takes to set up the JavaScript ecosystem. We make a "main.cljs" file, and that is the top of the application. It's a signpost. "Hey! Hey! Look here first!" The tab's not going to go away, so all we need to do is start up all the event listeners because JavaScript is a very event-driven language. I want "main" to be a table of contents of everything that matters in the app: the views, the routes, the URLs, browser hooks, web sockets, etc. The worst kind of "main" is no "main" at all. There are frameworks where you make a whole bunch of separate files for each of your routes. I love how many times we said the word "react" in this episode. It's all very event driven. That's just the model of the whole browser. It's the water that you swim in, so you must swim the right way in order for the application to succeed. Reactive cycle User Interaction → Event → Callback → Reactive Model → Re-Render Kinds of frontend components Navigation (eg. browser history) Router Settings State holders (eg. app state, reactive atoms) Pure models Browser integrations (eg. camera, microphone, GPS, notification API, fullscreen API) Render Links "Reentrant Coding" Series Ep 114: Brand New, Again Ep 115: The Main Event Clojure for static sites Stasis Powerpack ClojureScript for frontends shadow-cljs Figwheel Reagent…

F

Functional Design in Clojure

1
Ep 115: The Main Event 28:42

1 year پیش28:42

28:42

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "the main function" . We look for a suitable place to dig into the code and find an entry point. Our discussion includes: We're diving into a Clojure project. Where should we start? First up: the backend. Why start with the code? Why is "main" so important? What's the problem with lots of entry points? What should a good "main" do? How does a good "main" help you find things? What is a good component system? The difficulty of understanding inter-component dependencies. What is a "table of contents" for your program? Random access reading. What are common kinds of components? What is a pure model? How do pure parts and side-effecting parts all come together? Selected quotes Be friendly to people that come into your code base. I didn't trust myself as much as I should. "You just start at the top and write from the top to the bottom." "That's how I code everything. It's just one really large file." So all parts of your code are reachable from "main"—or should be. A great "main" is where you can see all the major parts of the application and how they fit together with each other. A terrible "main" is a system that doesn't have any "main" at all! It has a thousand different entry points that are all over the place. A great "main" is very compact. You can scan it. It's a very high level recipe of what's going on. Component has a system map. You can just look at the data structure and see all of the different components—the major players. The alternative is components that declare dependencies on each other. It's a kind of nightmare. Everything running independently, calling or referencing each other. What's using what? What's calling what? How does information flow through? When dependency information gets spread all over the place, you have to go to all the different places to even understand what you need. Having it all in one place is essential for understanding. It really helps when you can see the interdependency between things really easily. Each component should only get what it actually needs. It shouldn't just get the whole map of every dependency. Common kinds of components: shared resources, integrations in, integrations out, pure models, state holders, and amalgamations. It all comes together in the amalgamations. Pure models are the core of the application. They are higher level than just data manipulation. All of the actual nuts and bolts is in the pure models, and that makes the components relatively light. The goal is to make the pure model as fat as possible without introducing any non-determinism, AKA side effects. Amalgamations: it's where the "in", the "out", and the pure model all come together—where the real work gets done. The amalgamation components end up being at the middle of the application. They're a kind of orchestrator. Common kinds of components shared resources integrations in integrations out state holders pure models amalgamations Links Component Ep 097: Application of Composition . We discuss having table-of-contents style sections in your code. It helps with finding things and forming the big picture.…

F

Functional Design in Clojure

1
Ep 114: Brand New, Again 16:08

1 year پیش16:08

16:08

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "what's old is new again" . We find ourselves staring at code for the first time—even though we wrote some of it! Our discussion includes: Christoph re-opened his own code base after 14 months! Nate joined a software team with a large, legacy code base. Why does the problem of "fresh eyes" matter so much? How is my past self going to treat me? Are the past teammates going to help me? Why you shouldn't rely on memory. What's involved in really understanding a code base in order to make a change? The frustration of being unproductive. How is an app put together? What's the flow of information? How do you really know if your code is comprehensible? Selected quotes It's always fun to start something new. You don't have to worry about all those other things from the past! I was a little nervous about that other guy who made all the code: me fourteen months ago! I wasn't sure how good of a teammate he was going to be. The word "legacy" is like a big bad word, but I feel like it should be a badge of honor! It's software that is running right now and paying your salary! You should have a reverence for code that exists and is running. At some point in time, you or a new team member, is diving into a code base with fresh eyes. It brings up so many important issues that we face as developers. We spend so much time reading code and forming mental models about what is going on. A fundamental challenge in software development is understanding, comprehending and reasoning about the code base. Comprehending and reasoning about the code is one of the primary drivers behind the "why" of a lot of the so-called "best practices" of the industry. Why do you write tests? Why do you write documentation? Why do you try to have a good design in your application? There's this constant learning that we have to do, and so try to make that easier. He moved on to better projects in the sky. We've lost him to a better project in the Cloud. He moved to a better project upstate! It's easy to say: "We have great documentation! Our code is super readable! Decoupled? Absolutely! Our pure data models? Totally comprehensible!" It's easy to say that, but you really find out if those things are true when somebody new joins the team or when you have to revisit the code after a long time. Always trying to teach someone else about your code. There's always some future person. What can we do now as we're setting up the situation for new people in the future?…

F

Functional Design in Clojure

1
Ep 113: Highlightify! 27:13

1 year پیش27:13

27:13

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "highlight highlights" . We highlight the highlights of the Sportify! series. Our discussion includes: Sportify: The Highlights! Why is Clojure so well suited for situated problems? Why should you work with real-world data ASAP? How does REPL-driven development move seamlessly from exploration to implementation? Don't invent your domain model, let it emerge! How do you grow the application domain? How does Clojure help facilitate bottom-up development? Why is early integration just as important as exploration? How is reliability related to frequency of use? How does pureness affect reliability? What is the think-do-assimilate pattern? Are pure functions even needed in I/O heavy applications? How do you shift a code base from imperative to pure? Why is data-centric simpler than behavior-centric? Selected quotes "Let's put it all together into one context to rule them all and in the darkness bind them!" "But we're trying to spread the light of sports highlights across the Internet!" There's nothing like actually seeing real data come from real APIs. No amount of talking to your boss or talking to the intern or reading documentation can replace what you get from touching the real-world situation. Clojure helps you figure out how to bring the pieces together because you can just run the pieces in an ad-hoc way. You can just work on each of the parts without having to unify them into some kind of global proof system that's being foisted on you by static analysis. It's like whiplash-driven development: you're moving so fast, you have to take a break just to take a breath! The bottom-up way of constructing in Clojure has two properties: you're grounded in the real world, and you're just making what you need as you go along. It's very efficient. Some of that code is going to find its way into your final working solution. You're always making progress. You're always grounded in reality. You're just building what you need as you go along. It's not wasteful. It's very iterative. Very lean. Always forward motion. If your system exploration is in Clojure, you can cross information streams a lot easier than if you're using separate tools. In Clojure, it's all data. You can just hand data back and forth. You're not only discovering the properties of each information silo you're working with, but you're discovering the properties of how that data might fit and merge together. It's a grounded, incremental process for each of the parts, but also as the parts come together. Sometimes you don't know what the final solution is going to be even though you have all the necessary parts. It's greater than the sum of its parts. It isn't until you start running things over and over more frequently that you begin to discover the smaller percentage reliability issues. The way you increase reliability is by minimizing things with side effects and maximizing things that are pure. You're learning, at every step, what point needs more reliability. The more pure data you have, the more visibility you have. The more pure functions you have, the more testability you have. So, reliability and pureness are definitely related. It is amazing how much opportunity there is to move things into pure functions. The actual fetching or querying of the thing ends up constituting a pretty small part of your application. Working with the data tends to dominate. The think-do-assimilate pattern allows you to maximize the testable surface in your application by factoring all the I/O out. You start minimizing the I/O parts because your domain has emerged. Links Sportify! series: Ep 101: Sportify! Ep 102: REPLify! Ep 103: Explorify! Ep 104: Assembleify! Ep 105: Codify! Ep 106: Robustify! Ep 107: Idempotify! Ep 108: Testify! Ep 109: Extractify! Ep 110: Contextify! Ep 111: Loopify! Ep 112: Purify!…

F

Functional Design in Clojure

1
Ep 112: Purify! 33:20

1 year پیش33:20

33:20

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "pure data, pure simplicity" . We loop back to our new approach and find more, and less, than we expected! Our discussion includes: Sportify reaches the end zone! How can you make software extremely reliable? How can you keep logic pure when it depends on the outcome of I/O? What is the "think, do, assimilate" pattern? What is a "context"? What information should it contain? What is an "operation"? What information should it contain? Why does "structural sharing" matter? When to use multimethods vs case statements. The benefits of reducing functions. Testing with mocks vs data. Why test with real-world data? How does minimizing I/O make your code testable, inspectable, and repeatable? How to support any process conditional. A general way to model any process. Selected quotes Let's get some Hammock Time. We're big fans of Hammock Time! We make a function for each of those: think, do, assimilate. A function to figure out the next thing, a function to do it, and a function to integrate the results back into the context. ("determine-operation", "execute-operation", and "update-context".) It's not a single point of failure! It's a single point of context. Where you have a binding in a let block, you have a key in a context map. There's a symmetry there. You can make the operation map as big, fat, and thick as you want, so "execute-operation" has 100% of everything it needs for that operation. The "determine-operation" function can decide anything because it has the full context—the full world at its disposal! Clojure has structural sharing, so all the information is cheap. We can keep a bunch of references to it in memory. We're not going to run out of memory if we keep information about every step. The "update-context" is a reducer, so we can make a series of fake results in our test and run through different scenarios using "determine-operation" and "update-context". We're able to test all of our logic in our test cases because we can just pass in different types of data. Your tests are grounded in reality. They're grounded in what has happened. We've aggressively minimized the side effects down to the tiniest thing possible! Data is inert. Mocks are not. Mocks are behavior. You can just literally copy from the exception and put it in your test. There's no need transform it. It is already useful. It's very testable. It's very inspectable. It's very repeatable. It creates a really simple overall loop. You want those I/O implementations so small and dumb that the only way to mess them up is if you're calling the wrong function or you're passing the wrong args. Once it works, it will always work, and you no longer have to test it. We need to build into context every little bit of information we need to know to make a decision. Context takes anything that is implicit and makes it 100% explicit, because you can't get data across the boundaries without putting it in the context. You have no option but to put everything in the context, so you know everything that's going on. We're in this machine, and there's no exit. We're on the freeway, and there's no off-ramp. We're in the infinite loop! How do we know we're done? How do we know we're done? Links Sportify! series: Ep 101: Sportify! Ep 102: REPLify! Ep 103: Explorify! Ep 104: Assembleify! Ep 105: Codify! Ep 106: Robustify! Ep 107: Idempotify! Ep 108: Testify! Ep 109: Extractify! Ep 110: Contextify! Ep 111: Loopify! "Hammock Driven Development" by Rich Hickey core.async Portal…

F

Functional Design in Clojure

1
Ep 111: Loopify! 28:54

1 year پیش28:54

28:54

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "trying again" . We throw our code in a loop, and it throws us for a loop. Our discussion includes: Sportify continues! When is it time to stop developing? How do we handle retries? What if you need to recur from catch ? How do we recover mid-process? Where should the recovery logic go? Is there a way to get all the critical context at the same level? What should you preserve across a recur ? What does it mean to be "loop native"? What is the basic structure for any automation? What is a "single application state"? Selected quotes It's a lot like having a project on a workbench. You have all of the tools and all the information laid out before you on that workbench. Nothing is tucked in a drawer or inside a cabinet. That's a very important lesson for any developer: you can always stop—at least after it's working. Nothing in the world is solved except by adding another level of abstraction. I was not expecting that level of mutation! I was expecting a Kafka log written in stone! The positive is it has everything. The negative is it has everything. We would like more loop-native code inside of our cloud-native application. Are you suggesting that just because we can, it doesn't mean we should? We're programmers! If the language lets us do it, it must be a good idea! One of the reasons why I like Clojure is because it specifically tells me that I can't do some things that are bad to do. All of the context is in one map. It has everything in it. One map to rule them all! Might this be the fabled "single application state"? We have the thinking function, the doing function, and the assimilate function. Links Sportify! series: Ep 101: Sportify! Ep 102: REPLify! Ep 103: Explorify! Ep 104: Assembleify! Ep 105: Codify! Ep 106: Robustify! Ep 107: Idempotify! Ep 108: Testify! Ep 109: Extractify! Ep 110: Contextify!…

F

Functional Design in Clojure

1
Ep 110: Contextify! 23:22

1 year پیش23:22

23:22

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "gathering debugging context" . Our downloads fail at random, but our dead program won't give us any answers. Our discussion includes: Sportify! Now, with even more exceptions! How do you know the state of the application after it has already died? Where do you catch the exception? What do you log and where? How do you get the context to where you need it? What place has all the important context? Where does the pure logic belong? How can you keep I/O functions ruthlessly simple? The beauty of fully described operations. Selected quotes We just need to fill out our support ticket and say, "Hey! Fix your service!" It couldn't possibly be our code! So there is an error happening, but what happened just before that error? It is dead. There's no way to ask it any questions. It will not give us any answers. The only way to know what the program was doing, is to know what the program was doing. If you're trying to figure out what the program was doing by reverse engineeringing it, you're going to get it wrong. I love hiding side effects with macros! That's one of my favorite things to do in Clojure! It makes me feel like I'm using Scala again! We don't want the I/O function to do any thinking of any kind. It's a grunt. We fully specify the bits it needs to know. It's 100% a boring outcome of what we passed into it. Those I/O functions end up being ruthlessly simple. They're often just one line! We remove the thinking, so we remove the information. It's not because we don't like pure functions. We put them in a place where we can have all the information in one place. We're getting to the point where our let block is getting really long really—maybe too long. We're really letting ourself go! Links Sportify! series: Ep 101: Sportify! Ep 102: REPLify! Ep 103: Explorify! Ep 104: Assembleify! Ep 105: Codify! Ep 106: Robustify! Ep 107: Idempotify! Ep 108: Testify! Ep 109: Extractify! Cognitect AWS API…

F

Functional Design in Clojure

1
Ep 109: Extractify! 26:35

1 year پیش26:35

26:35

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "separating data from I/O" . We need to test our logic, but the I/O is getting in the way. Our discussion includes: Bugs in our Clojure code? What?! You must mean Javascript. When does Hammock Time not help? How logic grows and expands. Why you should test your logic. What parts do not need testing? What's a functional approch to working with APIs? How do you separate out logic for complicated I/O sequences? I/O testing without mocks. Why do we create our own model when Java gives us a class model already? The problems of built ins. Selected quotes We're using Clojure. Everything should be perfect, right?! I love Hammock Time for figuring out hard problems, but in this case, I think we have a simple problem of testing. You got to have the right amount of celebration after all those "line crossings" and "goal scorings" and stuff. We're doing a relatively simple process: we're downloading things and compiling them together into a file. But, it's amazing just how much logic is all throughout this process. As soon as you make a process, there's always going to be people who want to do it differently! If experience is any indicator, you always need more information. One of the reasons why you test is, when you make this kind of logic change, you want to make sure that everything continues to function. You need to write tests so that when you make future changes, your old self is there sitting right next to you making sure that the old use cases are all covered, so that you only have to think about the new use cases. With REPLing, you're figuring it out. With tests, you're locking it down and making sure that you have coverage in different situations. Our biggest obstacle here is that logic and I/O are mixed up together. Wait! Wait! We want to test our code. We don't want to spend our life writing code. Did you write the mock correctly? How do you write a test for the mock? I think we need to completely pivot our approach here. The problem is that we have I/O, logic, I/O, logic, I/O, logic. We have those two things right next to each other. What we should do instead is completely invert our thinking. Let's gather information and then we can do pure logic on that data. Separate those two things. We're going to extract from those POJOs. [Groan] I've got to use these terms every now and again or else I'm going to forget them all. So we do an I/O call, collect information, and create our own internal representation. We just need a few bits of it, so we create a working representation of that. It's our representation. It's our program's way of looking at the world. Craft the different scenarios in data that represent all the real life situations we found. One of the problems of using built-ins is: what parts matter? We're accreting working information into a larger and larger context. You're setting the table with all the pieces that are defined in your working world and then creating unit tests in terms of those. The world was like, "Hold my beer!" Links Sportify! series: Ep 101: Sportify! Ep 102: REPLify! Ep 103: Explorify! Ep 104: Assembleify! Ep 105: Codify! Ep 106: Robustify! Ep 107: Idempotify! Ep 108: Testify!…

F

Functional Design in Clojure

1
Ep 108: Testify! 22:42

1 year پیش22:42

22:42

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "testing around I/O" . We start testing our code only to discover we need the whole world running first! Our discussion includes: How do you unit test an I/O heavy process? Should you be REPL-driven or test-driven? What is the REPL suited for? What are tests suited for? What do you need to know to figure out the bug? How can a purely functional language help with testing? Techniques for factoring out pure logic. What is an extraction function? What is an ingestion transform? Outside data models verses "internal" or "working" models. Code smells when working with external data. Where can you use schemas in your code? Selected quotes The tracer bullet misfires every now and again. Now you're going from a tracer bullet to a silver bullet—apparently trying to solve all the problems at once! The REPL lets you figure out the basics of the process and your own way of thinking about it and modeling it, and the tests let you start handling more and more cases. Exploration early, testing later. Are you just supposed to log everything all the time? Always run your code with a profiler attached? If you look between each I/O step, there is pure connective tissue that holds those things together. We remove the logic and leave just the I/O by itself. With pure functions, we don't have to worry about provisioning the AWS cluster for the tests to run! It's really tempting to use the external data as your working data. What is the data that this application reasons on? By creating an extractor function, you pull all of the parts that matter into a single place. It returns a map for that entity that you can reason on and schema check. We've distilled out the sea of information into a drinkable cupful. We've gone from the mountain spring to bottled water. I guess you could always take all the raw data and shove them off in an Elasticsearch instance for massive debugging later—in some super-sophisticated implementation. Not how do we accomplish it, but how do we test it? Links Sportify! series: Ep 101: Sportify! Ep 102: REPLify! Ep 103: Explorify! Ep 104: Assembleify! Ep 105: Codify! Ep 106: Robustify! Ep 107: Idempotify! Related episodes: Ep 027: Collected Context Ep 029: Problem Unknown: Log Lines…

F

Functional Design in Clojure

1
Ep 107: Idempotify! 31:53

1 year پیش31:53

31:53

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "handling endless errors" . We discover when giving up is the way to get ahead. Our discussion includes: Finishing up our Sportify tracer-bullet implementation. What context can you put in a file name? Tricks on working with time information. Trying to "fix" a situation vs adding more context. What is a "positive representation" of a problem? What are "fundamental" versus "derived" facts? What information tends to be more stable? What information should you try to co-locate? How do you handle resources that are not currently available? How many times should you retry? How long should you wait between retries? How do you figure that out? What is idempotency? How can it help? At what point should you stop trying to handle errors? Can you have too much automation? How can you troubleshoot intermittent problems? Selected quotes We don't need a time zone offset because we know it's in UTC! In the spirit of building up the language to meet our domain, we can write a pure function! We want a deterministic way to go from this kind of common information into all the other bits of derived information. I was really hoping that we would finally be done with the errors and we could just get a highlight clip, but yet again, the world has conspired against us to make our life difficult as a programmer! Hold on! Hold on! The first thing we should do is run our process again because maybe the error will just go away! The problem does not go away in this instance. The problem just goes back to being hidden! The frustrating thing about programming is that code will do exactly what you told it to do. Clojure is already positive about nothing, that's what we call nil , so why not be positive about bad stuff too? Now I'm personifying the function as myself! It's better to reference information by its main identifier as opposed to some derived identifier. The context is all over the place: some context in memory, some context in the file system, and some intermediate context in the imperative function. That's the last problem we encounter, right? You never know what the world's going to throw at you! Why don't we just run it again? Let's just run it again! Maybe it'll be ready now... Maybe now... Maybe it will be ready now... And then we hit another error which is: the MAM rejects us for making too many requests! How long do you wait? Should you back off? That adds a lot of complexity to the code at that level. Because adding idempotency increases the complexity of that part of the solution, we only want to add it where it's necessary. The error condition is happening right now, so let's write the code to fix it right now. That's the way automation is at some point in time: a human needs to do it. You can find yourself in a situation where you're trying to do too many automatic things. Is it really worth doing? You cannot solve every possible permutation once you're interacting with the real world. It's okay to put up the guardrails, and if I'm outside them, robot me throws up my hands and says, "Human please!" But what happens when you run it again and the clip is ready? Now your retry logic doesn't get tested. Links Sportify! series: Ep 101: Sportify! Ep 102: REPLify! Ep 103: Explorify! Ep 104: Assembleify! Ep 105: Codify! Ep 106: Robustify! Related episodes: Ep 027: Collected Context Ep 029: Problem Unknown: Log Lines…

F

Functional Design in Clojure

1
Ep 106: Robustify! 30:37

1 year پیش30:37

30:37

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "building up reliability" . We push our software to reach out to the real world and the real world pushes back. Our discussion includes: Progress on the Sportify implementation. How do you keep development rapid as functionality grows? How can you learn and iterate quickly as you encounter failures and edge cases? The real world vs the portrayed world. ("The map is not the territory.") How to handle configuration during development. How to handle large downloads. What to do with I/O exceptions. How do you make software reliable? Where should you invest your effort for reliability? What learning is the most important in the tracer-bullet phase? How do you handle intermediate artifacts? What are different ways to handle failure? How to work with temp files. Exception handling tips and tricks. Why "what ifs" are an appealing trap. What type systems cannot help with. Finding joy in errors and failures. Selected quotes This is very incremental. We are rapidly accreting functionality. We're bringing it together with a very interactive REPL-driven way of getting things done. There's no reason why we would ever need to modify this code ever again because everything will go smoothly! I'm sure nobody will ever change their mind on functionality and nobody will ever make a mistake in any of the other systems either! You always have to deal with the uncertainties through time, not just the uncertainties of requirements. One of the great things about this interactive REPL-driven way is that we are exploring the real world. We're not exploring somebody's documentation or somebody's article or somebody's representation of the real world. We're actually interacting with real systems, and we're looking at real data, and we're figuring out the real situation. We're not trying to make the ideal version for production. We are trying to get a fully automated solution end to end to understand all the specific situations we have to handle. That S3 function is built on a tower of abstractions. Some of those abstractions involve the network, and other ones involve other companies. There's a variety of reasons why those might fail—whether for geopolitical or network-based reasons. The human retry loop is a completely valid solution at this point in time. It's actually a valid solution for a lot of things. Every time a human has to retry (and the human being is us) we learn every time. We're learning how the systems fail, and that's just as important as the happy path. Over time, we're going to accrete more and more reliability in the system by handling more and more things. Deleting the temporary files is a return to known state. It's a pretty harsh return to known state, but it is a return to known state. The initial state of having nothing is a sound state to return to. I/O is the greatest source of failures when you're automating processes. Nobody asked your program for permission to turn off the power. The boss man says, "Make some more! Make some more!" Let's reduce the recovery time as opposed to trying to avoid the need to recover. We can convince ourselves that work is needed because we have evidence of failure over time. We're growing functionality on demand as needed. It's very lean. We're building the right software just in time. Not only are you iterating quickly, so it's not a long time between each change in each rerun, you're also building the right thing every time. It is in the realm of the world, not in the realm of what you think the world is going to do. The actual world is there. The situation we're in is the real world. It's not something that could happen or maybe happen or "what if" happened. I/O is the source of pain in our lives, but it's the source of actually making useful software, so it's worth it. Even if you're in a programming language like Haskell that tries to do proof systems around your logic for handling I/O, it still can't save you from the fact that I/O is going to blow up on you! I/O failures happen. What happens if you need to retry the retry and then retry that retry? There's only so far you can go with this imperative assembling of the application. We're letting the reality of our situation dictate where we apply our effort. We still have learning to do. Well, that was exceptionally fun! We're still having fun even though we're encountering errors. Both of those things can happen at the same time! It's just delightful to see progress. You're always feeling progress! That's a big goal: feel the wind at your back! Links Sportify! series: Ep 101: Sportify! Ep 102: REPLify! Ep 103: Explorify! Ep 104: Assembleify! Ep 105: Codify! Cognitect AWS API…

F

Functional Design in Clojure

1
Ep 105: Codify! 25:42

۲ years پیش25:42

25:42

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "building up a solution" . We grow beyond our REPL-driven pieces toward an end-to-end solution. Our discussion includes: How to move from exploration to implementation. The overall process for Sportify highlights. How our REPL exploration set us up for implementation. What is a "tracer bullet" implementation? How is it useful? What is "imperative decomposition"? What is "imperative composition"? How to handle surprises in the data. The first code solution for an I/O-heavy process. How to build up a solution incrementally. Running and testing the intermediate process. How to visualize intermediate data. What approach is both "coding first" and "coding light"? How does Clojure allow a domain to speak for itself? Selected quotes The learning is complete. Now it's time to get to the programming! We didn't sign up to be a robot. We signed up to be a programmer. We want fighting teams. We're not going to have very many highlights if it's the Badgers versus the Doves. A tracer bullet is a minimalist solution where we try to get something working end to end. It's interesting that you would say "imperative decomposition", because in this case, we have the parts, so we are doing an imperative composition. We're putting them together. You get your learning, and you get a little bit of code out of it at the same time. Sure, that code may not be what you want to use in production, but you certainly have more actual code to work with than you did if you just opened up your database explorer and ran SQL statements. Just because it's a silver bullet doesn't mean all human intervention is no longer needed. A silver bullet has to be fired by someone! We're growing a function a piece at a time. So by naming that and giving it a function name, it makes it more readable. It helps document the information. It's like a mini language here in the let . ... It makes this process—that you're now documenting in code—a little more readable. You can launch this whole thing using a comment block! You're exploring your way toward a solution. Even though it's very imperative in this case, you're just continuing to explore your way towards the solution as you build it up. You're on your second rewrite of the code. You're not writing this code for the very first time. You're writing it for the second time, which means you're going to be picking variable names and function names better than the first time because you understand the concepts. You're not learning the concepts and writing the code at same time. Exploration was important, but this second step (of making the code again) is valuable, because you're learning what level of abstraction you want. With command line clients and browsing tools, you still learn a lot about the domain, but you don't learn a lot about how you want to represent it. It's a coding-first approach, but it's also a coding-light approach. Clojure doesn't make you model the universe in a proof system that you have to try to get right and revise and revise. Links Sportify! series: Ep 101: Sportify! Ep 102: REPLify! Ep 103: Explorify! Ep 104: Assembleify!…

F

Functional Design in Clojure

1
Ep 104: Assembleify! 35:03

۲ years پیش35:03

35:03

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "exploration cessation" . We realize we're done exploring when all of the pieces fall into place. Our discussion includes: We fiddle at the REPL to explore Sportify! What is random access exploration? What are you learning as you explore? How to interact with AWS and S3 Separating out operation instructions from execution Maximizing side-effect free code Handling credentials when fiddling at the REPL Visualizing data when fiddling at the REPL Practical tips on handling data in your fiddles Why use the REPL when you have command line tools? How does exploring via the REPL help you figure out the structure of your application? How do you know what to factor in your application? How to run external commands. How do you transition from exploring to application development? Why is hands-on experience with the data indespensible? What are "exploring abstractions" vs "application abstractions"? What is "data variance"? Why is it important? Selected quotes That's how I approach problems: you just keep going. Keep moving. A fiddle is like a random access REPL. A fiddle is just a file. We can put all the different bits of information—including data—in that one file. There's only one place to go back and look at when we pick up the project the next day. Well, the MAM actually doesn't store the media. I feel like we were shortchanged! It's the manager. It doesn't do the heavy lifting. When else have you seen the manager do the heavy lifting?! Will our troubles never cease?! There's constructing the request and then there's doing the request. We're here to explore. We're not here to create bike sheds with bike sheds inside. Yes! This is not the place for abstractions. We're building up enough language to help us with our exploration. Only write the functions that you need to help you learn more. As soon as you've learned enough, stop writing functions and move on. The point is to keep learning, not to keep abstracting. You can use the command line, but it's worth doing in the REPL, because you're starting to actually explore the library too. Utilitarian tends to win in the end. Things that let you get things done quickly. Those solutions are great solutions. It's good to fiddle because you can play around with it. You can write it the wrong way four times, and get those out of your system before arriving at what you want to use. The fiddle is here to help you figure it out. We were doing all this exploring, and we stumbled into doing some actual work! There's no better way to know you've arrived at the end of your exploring then when all of a sudden, the thing you're trying to do, is now finished! Over the course of your exploration, you go from exploring to doing actual work. There's no seam between the two activities. Exploring dwindles down as you know more. Your fiddle is turning into your recipe: a semi-automated way by hand. It's like you've stumbled into a working program. You're still learning as you go through the process again and again. You're always learning. You cannot overstate how important it is to experience the variance of the data by hand. You think, "Oh! I see the pattern!", and then it's example seven that blows up your pattern. Then you think, "Oh, but now I know the pattern!", and then example fifteen blows it up. You're getting a lot of direct experience with the process. That's going to help you make better abstractions for the application. You're going to learn it sometime: either now or in the future. It's better to learn it now when you're in exploring mode than later when you're on the hook and your boss is breathing down your neck! We've sent the asynchronous notification back to the work giver via the Outlook message queue. It's fun to do the first 10, but after that, you probably want the computer to do all the heavy lifting for you. Links Sportify! series: Ep 101: Sportify! Ep 102: REPLify! Ep 103: Explorify! Cognitect AWS API Amazonica ffmpeg clj-media Babashka babashka.process babashka.fs…

F

Functional Design in Clojure

1
Ep 103: Explorify! 32:15

۲ years پیش32:15

32:15

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "exploring new data and APIs" . We peruse APIs to uncover the data hidden beneath. Our discussion includes: More exploration of Sportify! How to explore a database from the REPL. How to work with SQL rapidly. How structural editing speeds things up. Building up the vocabulary of exploration. What should the process of exploring via the REPL feel like? What is a Media Asset Manager? How do you make sense of a brand new JSON API you've never worked with? How do you work with a poorly documented API? What problems are complected for HTTP requests? How to use composition to speed up API exploration. Dealing with the non-linear nature of exploration. Saving results and experimenting with them. How can you speed up experimentation on large data sets? How to handle excessively normalized APIs. Selected quotes "This is a situated problem. It's a real-world problem. Like many real-world problems, there are parts we control and parts we don't control. We have to figure out those parts." "What do we need to know to figure out the information?" "I like to start with the way people naturally talk about these things. If you start building up your language, it helps to describe things in either the innate input the system must have or the way people talk about it." "[Honey SQL] will do all the interpolation for you. It's just wonderful! That one feature alone would be enough, but there's plenty more." "That's a great thing about Hiccup and Honey SQL and other formats that use Clojure data structures to describe things: you get the power of structural editing." "It's yet another example of building up the vocabulary of the system so that you're able to talk about it at a higher level." "Practicality is the name of the game here. Imagine you're exploring in a forest and you're trying to figure out what you want to put in your backpack to take along with you. You don't want to take a lot of heavy, complicated tools. You want to only bring along the stuff that is actually useful to you!" "You only need to build up what is necessary to keep exploring because that's the point. The point is to learn. The point isn't to pre-optimize and make the abstractions." "My activity is centered around this fiddle file as I'm exploring." "We have all this cool information, but we're not making database table highlight reels. We're making sports highlight reels in Sportify!" "How do you make sense of a brand new JSON API that you have never dealt with before?" "By using it." "This is a good opportunity to mention we have a series called 'Web of Complexity'. The title should give you a sense of what we think of HTTP in general." "The name web should already be a bad omen! We never use the name 'web' in a positive sense anywhere else." "Perfect time to use our nonlinear history, aka the fiddle!" "We've made several really composable ingredients that we're now mixing together as we're learning more about the system." "Most of my time was spent sifting through the data, not actually making the calls!" "I'm communicating to myself tomorrow, because I want to forget all this context, go home, do something else, not think about work, and come back to work and pick it all up again." "One of the benefits of using Clojure to explore is you have a full, rich programming language—one with a full syntax including comments. By doing it all in one fiddle with these comments, you can pick it up in the morning." "Our lives are nonlinear. We get interrupted. We take a break. We have the audacity to go home and not think about work!" "It's like a workbench. We're laying out our tools. We're laying out our pieces. Our fiddle file is our workbench and we leave little post-it notes on that workbench to remind us of things." Example code Here is an example that uses the Cloudflare Streams API . It requires authentication, so we want to factor that out. First, define some endpoints to work with. Create pure functions for just the part unique to each endpoint. (defn check-status-req [id] {:method :get :path id}) (defn delete-req [id] {:method :delete :path id}) Then create a function to expand the request with the "common" parts: (defn full-req [cloudflare api-req] (let [{:keys [api-key account-id]} cloudflare {:keys [method path], throw? :throw} api-req] {:async true :method method :uri (format "https://api.cloudflare.com/client/v4/accounts/%s/stream/%s" account-id path) :headers {"Authorization" (str "Bearer " api-key) "Accept" "application/json"} :throw throw?})) For the purposes of illustration, the cloudflare configuration is: (def cloudflare {:api-key "super-secret" :account-id "42424242"}) See the full request like so: (full-req cloudflare (check-status-req "abcdefg123456789")) Which returns: {:async true :method :get :uri "https://api.cloudflare.com/client/v4/accounts/42424242/stream/abcdefg123456789" :headers {"Authorization" "Bearer super-secret" "Accept" "application/json"} :throw nil} Use [babashka.http-client :as http] , and call the endpoints like so: @(http/request (full-req cloudflare (check-status-req "abcdefg123456789"))) @(http/request (full-req cloudflare (delete-req "abcdefg123456789"))) Note, that in a REPL-connected editor, you evaluate each form, so you can see just the check-status-req part, or move out one level and see the full-req part, or move out one more level and actually run it. That lets you iron out the details to make sure you have them right. Finally, you can make some helpers to use in the REPL or imperative code. The helpers stitch the process together: (defn request! [cloudflare req] (-> @(http/request (full-req cloudflare req)) (update :body #(json/parse-string % true)))) (defn check-status! [cloudflare id] (request! cloudflare (check-status-req id))) (defn delete-stream! [cloudflare id] (request! cloudflare (delete-req id))) They can be called like: (check-status! cloudflare "abcdefg123456789") (delete-stream! cloudflare "abcdefg123456789") The helpers should do nothing except compose together other parts for convenience. All the real work is in the pure functions. Links Ep 014: Fiddle with the REPL Ep 063: Web of Complexity Sportify! series: Ep 101: Sportify! Ep 102: REPLify! next.jdbc Honey SQL Babashka http-client clj-http http-kit Portal Nippy…

F

Functional Design in Clojure

1
Ep 102: REPLify! 28:11

۲ years پیش28:11

28:11

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "using the REPL to explore" . We find ourselves in a murky situation, so we go to our REPL-connected editor to shine some light on the details. Our discussion includes: Diving into the world of Sportify! How do you get started when diving into a new automation problem? The exact workflow we use for exploring existing systems. What is a "streaming specification"? How can the REPL help us explore? What does it mean for an editor to be REPL-connected? What is a "fiddle file"? What is a "fiddle workflow"? The first things we do when we start a brand new project. Switching from Leiningen to Clojure tools. Options for where to put your fiddle files. How to run a single Clojure file. What goes in our very first fiddle file? How do we build it up? How we handle configuration when we're fiddling. How to get connected and explore a database from a fiddle. Tips and tricks for exploring a database from a fiddle. Using defs to help work with live data. How long does it take to get started? Selected quotes: "We often want to see highlights more than we want to see the game." "Like so many things with interns, there is really low supervision, so no one's really worried about how we get it done." "Having something done is better than not, so the bar is 0." "It sounds like a natural integration with an asynchronous message queue called 'Outlook'." "I like to call it 'streaming specification'. I'm not going to tell you everything. I'm coming up with this off the top my head." "We call them situated problems. They're problems that are set in the real world, so you need to know about all the things in the real world. It's a learning problem." "You say, 'the things we can't control.' I think of it as, 'the things that are foisted upon us!'" "We're going to find a way to use Clojure to solve this problem. I bet you didn't see that coming!" "The first thing I usually do is go for the completely correct and entirely comprehensive documentation that describes all of the different use cases that I might need, and in fact, has a sample code." (Ha! If only!) "It can be said that programming is debugging an empty file." "The first bug is my application does nothing!" "You want to get to your first rewrite as fast as possible, so write the messy version first. Then you can learn what you want the next version to be—even if the next version is also messy. The sooner you learn the better." "I'm starting to build up little pieces to help me explore." "We're going to figure it out interactively, using the REPL. We're getting our bearings." "Get in the code right away—hitting an external service as soon as possible—so we can begin to learn, so that we're solving the right problem instead of some problem of our imagination." "Once it's Clojure data structures, the whole world of Clojure opens up. All the power of mixing and matching and analyzing that data is open to you." "We're getting into the real thing. We're getting real data, real code. This is going to begin to grow up into something, but for now, we just want to see what's there—begin to explore—get some working code so that we can." "It's hard to imagine the workflow, because it's not a workflow that you do in other languages. If you haven't done this before, it's hard to just imagine it." "It's not just REPL first, but REPL-connected editor first, and there's a distinction." Clojure command line If you are unfamilar with the Clojure command line, check out the Deps and CLI Guide . You can set up global aliases in $HOME/.clojure/deps.edn . For example, this is one Christoph uses: {:aliases {:nrepl {:extra-deps {nrepl/nrepl {:mvn/version "RELEASE"} cider/piggieback {:mvn/version "RELEASE"}} :jvm-opts ["-server" "-XX:MaxMetaspaceSize=256m" "-Xmx1280m"] :main-opts ["-m" "nrepl.cmdline"]}}} Run it with: clojure -M:nrepl You can see more examples in Nate's dotfiles and in Sean Corfield's dot-clojure project. Links: Ep 014: Fiddle with the REPL Use #_ to ignore the next form Projects mentioned: tmux Calva clojure-lsp Libraries mentioned: next.jdbc Honey SQL…

F

Functional Design in Clojure

1
Ep 101: Sportify! 17:03

۲ years پیش17:03

17:03

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "introducing Sportify!" . We tackle a new application, thinking it'll be an easy win—only to discover that our home run was a foul, and the real world is about to strike us out! Our discussion includes: Introduce a new series! Sportsball! Sportsball! Going back in time. An overview of video production workflows. What is a media asset manager? How hard could it be? What could possibly go wrong? What are all the things we'll need to handle? What is a situated problem? Does immutability matter when most of the work is I/O? Code stability in Clojure. Selected quotes: "Clojure has made our lives fun, so we want to make your lives fun." "What do people love when they're watching sporting events? They love their highlights." "This is not a business problem. This is a sports problem, and sports problems are different." "Now this is where we're reaching the edges of reality, but just hang on. Come with us." "How hard could it be?!" "What could possibly go wrong?!" "But this MAM...do you have to be polite? Can I have the video ma'am?" "We've got to do the right amount. That's the hard part: the right amount." "Is there a fraught problem that's not situated, or a situated problem that's not fraught?!" "Situated, in my mind, is useful. I don't just want to heat the room up with my computer. I want to actually get something done!"…

F

Functional Design in Clojure

1
Ep 100: Thanks Overflow 35:29

۲ years پیش35:29

35:29

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "thankfulness" . We reflect on Clojure, the community, and how much we have to be thankful for. Our discussion includes: It's our 100th episode!!! Why we are thankful for Clojure and the community. The podcast origin story. Long haul programming. How Clojure keeps the mental burden low. Why we love immutability. Communicating Sequential Processes. Bad programming experiences that Clojure made better. Data-centric programming. What makes Clojure simple. The horror of unstable languages. Things we love now that we never imagined. Why we love structural editing and REPL-driven development. Developer flow and productivity. What is hammock time? Why does it matter? How does Maslow's Hierarchy of Needs apply to programming? What do scissors have to do with Clojure programming? We are thankful for you!!! Selected quotes: "I guess I can classify myself as an older developer now. It doesn't feel like it, but I've been doing it for a while." "Clojure has definitely reinvigorated my joy of programming, and Clojure has allowed me to experience that joy in nontrivial programs." "It's fun to whip up a prototype, but Clojure is the first language that I've used where, when the program starts getting really big, it's still a delight to extend" "I don't think I could estimate the number of hours I have saved in debugging race conditions and crazy nondeterministic behavior, because of immutability and CSP specifically." "Back in my Java 2 Enterprise Edition days..." "Oh, there's an old wound!" "Chasing after the latest and greatest was kind of fun, until you've been burned by a couple of tech cycles." "Because Clojure is so simple, you're left alone with your problem, so most of your headspace is your problem: the thing you're trying to solve." "REPL-driven development and structural editing is an entirely different way of making a program—a process or a flow of making a program—that for me was just life changing." "The REPL is so close at hand. It makes things so fungible. You can't help but want to use it for every problem that you have." "The REPL makes programming more tactile." "Not only are you flying through execution, you're flying through editing." "The state of being that I get into in Clojure code is just so delightful! It's so enjoyable!" "It changes the way you think about programming and the way you think about editing. You don't realize that your mind is being changed until you realize that you're so different than before." "You still have the same amount of productivity, but you will have spent more time thinking about the problem, so that you're doing the right edits." "When you're fighting your code base, because it's exploding in random places, it's hard to think about bigger concerns. You're just trying to make the bugs stop already! Clojure has gotten me out of all that." "I definitely don't have the corner on all the truth in the world. That's for sure." Links: Composition Series Ep 093: Waffle Cakes (first episode) Ep 098: Composed Learnings (summary episode) Talks, Books, and Articles "Beating the Averages" by Paul Graham "Practical Vim: Edit Text at the Speed of Thought" by Drew Neil "Running With Scissors: Live Coding With Data" by Stuart Halloway. Presented at Strange Loop 2018. Clojure for the Brave and True by Daniel Higginbotham Practicalli Projects Clojure , ClojureScript Babashka clojure-lsp Conjure for Neovim Calva Los Angeles Clojure Users Group…

F

Functional Design in Clojure

1
Ep 099: REPL Your World 25:28

۲ years پیش25:28

25:28

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "taking the REPL beyond your application" . We free our REPL to explore and automate the world around us. Our discussion includes: What are the different ways of working with the REPL? How can you be more productive by using the REPL? What is the connected editor? How to use the REPL beside writing code for your application. What is often missing from API docs. Moving from bash to Clojure. Using the REPL for exploration. What is a "fiddle" approach to using the REPL? What is it good for? Why should you use your editor for non-coding activities? How to save time when you're stuck with manual testing. Interacting with databases. When is the REPL better than a script? How a REPL is like a bash prompt, and how it's not. What supports the supporting activities of software development? Using the REPL as your application interface. Migrating data using the REPL. Why your REPL is a natural place for difficult to access resources. Why the REPL saves you from extra coding. Selected quotes: "We share because we care." "The connected editor is an interface to productivity." "I like to call it whiplash-driven development because it's so fast that you literally have no time between when you write the code to when you execute it. You're just blown back by the productivity!" "This is a Clojure Podcast, so I bet you know where this is going." "The REPL as a window into another system." "I actually wrote the API, so I know how it should behave...but not how it does!" "Isn't that one of the goals of your programming language and experience: to spend less time doing the things that are really mundane and repetitive and more time actually doing new things?" "I don't want to just let that go into the history file. I want to save it someplace more important." "I have all of the power of Clojure before the query and then after the query. I don't have to trick psql to write that data out somewhere so I can read it in my REPL. It's already there!" "It's not that there aren't other ways to do this. That's not the point. The point is that all of a sudden you realize there's a lot of interactive exploration and processing and task automation you can do from the REPL because you don't need to write a script to do it." "You can just execute a form, and boom! It's off. It's running." "But then, I was like 'Wait, there's got to be a better way!' You know, infomercial style." "It's functional programming, so we're going to talk about composition. It'll happen." "Whoa! The REPL could be a terminal! A super powerful terminal into a vast warehouse of data, and I can slice it and dice it all sorts of different ways and discover things." "The REPL is just a way of doing structured execution very rapidly and very flexibly." "With the REPL, every function or form is a potential entry point. You can have all kinds of entry points." "The malleability of it is its power." Links: REPL Series Ep 012: Embrace the REPL Ep 013: Connect the REPL Ep 014: Fiddle with the REPL Composition Series Ep 093: Waffle Cakes (first episode) Ep 098: Composed Learnings (summary episode) Talks mentioned "Running With Scissors: Live Coding With Data" by Stuart Halloway. Presented at Strange Loop 2018. "Data Science in Clojure" by Soren Macbeth. Presented at Clojure/West 2015. Projects mentioned Nate's tabl project Nate's guide: Developing Clojure in Vim (2023 edition) Conjure for Neovim next.jdbc Honey SQL…

F

Functional Design in Clojure

1
Ep 098: Composed Learnings 26:12

۲ years پیش26:12

26:12

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "composition is life" . We reflect on keeping the necessary mess at the edges so our core can be composed together with beauty and simplicity. Our discussion includes: Our reflections on the episodes in this series. The best concepts from the episodes in this series. How to separate data from behavior. Why zucchini bread is at the core of functional programming. What baking has to do with functional programming. How to get objectivity for better understanding. Why you should focus on trade offs. How Clojure and immutability steer you toward composition. An inventory of different kinds of functions. Why side effects ruin composition. How Clojure lets you focus on your problem. How composition works in the small and the large. The paradox of constraints providing more freedom. The cognitive load of functional programming vs object-oriented programming. Selected quotes: "This episode we thought we would take it and put it all together." "The last draft." "Functional programming is built on composition." "Object-oriented programming tends to have at least a little bit of behavior and data together, so you can't separate them if you want to make a new thing." "If you just want classes to be one thing, just data or just methods, all of a sudden you end up with something that looks a lot like functional programming." "Constraints breed freedom and clarity. If you can't reach other functionality through data, then data will never hurt you." "It takes a while to discover what is worth separating and what's fine to leave in larger chunks." "You have to use it." "So by moving [a problem] into a separate domain, you're able to see the essentials, the fundamentals, more clearly because you're not stuck in those ruts of thinking that you do without realizing." "I've been known to eat the same thing for breakfast every day, for weeks on end." "Clojure core plus immutability really steers you in the direction of composition." "'Transforms' is a little unfair of a category because that's almost all of functional programming." "Data cake. Delicious!" "I want to do something with this data that affects the world around me besides generating heat." "Side effects ruin composition because side effects are a second dimension that is difficult to compose." "When you write functions in the way that Clojure expects them, this whole world of core opens up, and the only thing you had to implement was your the code that was relevant to your domain." "There's almost nothing to Clojure itself. Most of my time writing in Clojure is thinking about my problem--not thinking about how I'm going to express it or how I'm going to model the object graph or any of that stuff." "I'm just left thinking about my own problem." "The key in all composition is a clear delineation and separation of responsibilities." "Because they're smaller, you can understand them at a glance." "With side effects, there's an infinite number of additional dimensions that are possible. You can't just worry about what the function says it does, you have to worry about all the other things that it could be doing!" "Side effects are like really advanced politics between hostile countries. There are the words that come out of their mouth, and then there's all the stuff that really matters that no one's talking about!" "Clojure is for the peaceful." "If you like to keep all the messiness at the edges so the core of your application can be composed together with beauty and simplicity, making your day wonderful, then Clojure might be for you!" Common kinds of functions: Predicates: provide truth values Reducers: take the current "state" and an input and return a new state Transforms: broad category that covers many kinds of functions Conversions: take one kind of thing and turn it into another kind of thing eg. parse-long Extractors: return part of a data object Mergers: combine data together Decorators/enrichers: takes a data shape and adds more into it while preserving the original data shape Side effectors: affects the world in a clear and specific way Links: Ep 093: Waffle Cakes Ep 094: Concrete Composition Ep 095: Composing Core Ep 096: Gaming Data Ep 097: Application of Composition…

F

Functional Design in Clojure

1
Ep 097: Application of Composition 28:33

۲ years پیش28:33

28:33

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "composing your application" . We get a handle on bringing I/O resources together in an application. Our discussion includes: What is a "handle"? Why is it called that? More puns! How do you handle I/O handles? How do you keep the pure and side-effecting parts away from each other? Managing and organizing application components. What is a framework vs a library? "Injection style" vs "singleton style". How far down the call stack should handles travel? Figuring out a start-up order and a shut-down order. How obtaining a resource can negatively affect composition. Separating the knowledge of finding a resource vs using a resource. The concept of an "application index". Why all this matters for long-term maintenance. Using feature flags. The two developers on every software project. What is the "band-aid test"? Selected quotes: "If a program does a bunch of computation, but it doesn't have any I/O, did it do anything?" "Handles inherently are going to hurt you. You don't even have a hold on the whole thing. You have a little bit that you can use to to access that external system." "I visualized an axe or a knife. You have a handle, but you have to watch out for what the handle is connected to!" "In Clojure, I think of frameworks as an integration between your code and the purpose of that framework as opposed to the foundational layer that glues your whole world together." "Even in OO, it's a smell if I/O handles travel too far and wide!" "Lots of assuming!" "Sometimes ad hoc is confused with flexibility. You didn't have to think about it, so it felt flexible, but it just happened." "Composition is about being able to use things in multiple ways." "If you are accessing a singleton state from a function, you are no longer side effect free." "Why would I ever save anything anywhere other than the place the application has configured as its database?" "Have you ever had a physical "dead tree" book?" "Where are all the components?" "With the index style, you have a clear delineation of all the parts of the software and how they're connected together." "Whenever you start using a library or framework, you are hopefully solving a problem, but you're also inviting complexity in. You have to be wary of what it allows you to do, and what it allows future developers to do." "If you pull that band-aid off, how many little hairs are going to come off with it? How badly is it going to hurt?" "One of the tests of composability is the surface area of contact. More is not better!" Links: Injection style: Component + component.repl , Integrant Singleton style: redelay , mount Built on Component: system Others: donut.system , Components in Biff Ep 063: Web of Complexity Ep 067: Handling Handler's Handles Ep 093: Waffle Cakes Ep 094: Concrete Composition Ep 095: Composing Core Ep 096: Gaming Data…

F

Functional Design in Clojure

1
Ep 096: Gaming Data 26:39

۲ years پیش26:39

26:39

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "playing games with data." We go back to start and play through a composition strategy to see where we land. Our discussion includes: The return of Tic Tac Toe. It's back! The biggest difference between object-oriented and functional composition. Can a game board play its own pieces? Isn't that the stuff of horror films? What does it mean to be "complected"? What's more important: data or behavior? Why separate data from behavior? The simple domain of batch processing. The dark magic of hidden side effects. What lasts longer: data or your application? Representation vs operations in an information domain. Common kinds of functions. How do schemas fit in? Where does I/O fit in? Rationale for organizing your namespaces. The horror of code organization in Java. The stability of pure information models. What is language imposed complexity? Selected quotes: "It's been a few years for us, but it could have been a few days if you're really on a binge on this podcast!" "The biggest and most important difference between object-oriented programming and functional programming is the relationship between data and behavior. Functional programming keeps those separate, whereas object-oriented programming likes to mix them together into classes." "When I want to play a piece, does the board know how to play pieces onto itself? Or, does a piece know how to place itself on the board?" "You can't just have the functionality sit on its own. It has to be attached." "I arrived at a point where I thought, 'What am I doing?!'" "All kinds of different scripts that operate on the same data file don't need to know about each other at all." "I'm just going to get the name attribute from the Person object, right? You don't know. Am I getting a field, or am I getting some magical overloaded getter thingy that does a bunch of side effects?" "Data is simple, and data will almost certainly outlast your application. Often, the data that your application deals with came from somewhere else in the first place." "This reminds me of the fun abomination known as object-relational mapping where you take the data that was there before your program and you pour it somehow into your objects, then you do something with it, and then you pour it back out into the database. Why is this such a hard problem? Because it's a hard thing to do! It's something that doesn't make sense." "In OO, there are way more dimensions of potential organization, so therefore, more things for a software development team to fight over." "The only place you can put functionality in object-oriented language is attaching it to classes." "In Clojure, you don't have a lot to fight over in your bike shedding sessions." "When you're talking about which object to put it in, you're not actually talking about your problem. It's a problem that is introduced by the tool choice, not by the domain of the problem." "Because they're separated, you get reuse. You get data composition and behavioral composition as separate kinds of composition--because they don't often happen the same way." "I think the lack of reusability comes in object-oriented languages, not functional languages. Because the problem with object-oriented languages is they’ve got all this implicit environment that they carry around with them. You wanted a banana but what you got was a gorilla holding the banana and the entire jungle. If you have referentially transparent code, if you have pure functions all the data comes in its input arguments and everything goes out and leave no state behind it’s incredibly reusable." ~ Joe Armstrong in "Coders at Work" Links: "Execution in the Kingdom of Nouns" by Steve Yegge Ep 002: Tic-Tac-Toe, State in a Row Ep 093: Waffle Cakes Ep 094: Concrete Composition Ep 095: Composing Core…

F

Functional Design in Clojure

1
Ep 095: Composing Core 17:37

۲ years پیش17:37

17:37

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "core and composition." We venture toward the core of a solution. Our discussion includes: Why do functional programmers like to talk about composition? What composition means in object-oriented programming vs functional programming. Must simple be simplistic? What is idiomatic composition? Common kinds of functions. Why functional programming is more like Legos than taxonomies. What Clojure already knows and what you have to teach it. Why Clojure allows you to put things together quickly. Why side effects can ruin everything. How Clojure encourages composition. Selected quotes: "I don't think you want me to read code to you. That sounds like an excellent way to fall asleep if you have insomnia!" "Maybe we need a white noise track of keyboard noise." "Composition in a functional language is much simpler because we just have functions." "Why have we spent so much time talking about composition when it's just functions calling other functions?" "These names are a little on the nose, but programmers aren't always the best at originality!" "Functional programming is essentially based on transformation. Everything is a transform at some point in time." "The only way to get anything done is to return a new thing." "Core is a backbone, a spine, and you're connecting a progression of things together." "Your job, as a functional developer, is to teach Clojure core about your domain." "They're all functions that remix very well." "Everything must be pure because side effects mess up this whole world." "Make functions small because functional programming makes it very easy to combine smaller functions into bigger functions. Build up your functionality from those pieces." Links: Ep 075: Merge With Fun Ep 079: Compose Thyself Ep 093: Waffle Cakes Ep 094: Concrete Composition…

F

Functional Design in Clojure

1
Ep 094: Concrete Composition 15:29

۲ years پیش15:29

15:29

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "decomposition." We help our code through a breakup so it can find its true colors. Our discussion includes: How to process text and add color. When is it time to decompose? Return of the box mix! How to separate transformation from I/O. How to use reducing functions to repeat computation. What makes code orthogonal? The woes of packing useful data into bits. Signs of complexity in the Java underlayer. What is the difference between isolation and decomposition? What are some natural points of separation? Selected quotes: "We are composing ourselves!" "I'm a terminal guy, so this is all in the terminal." "You can go look it up on the Internet. I'm not going to try to speak ANSI right now." "The box mix makes it really easy to make a cake, but it also constricts your ability to do more things with it." "By definition, any side effect is an orthogonal concern." "It's screaming for decomposition!" "Separating these out makes it so that it's easier to understand each part by itself." "We have our 'actual code' badge again as a programming podcast!" Links: Ep 093: Waffle Cakes Ep 031: Eager Abstraction Ep 033: Cake or Ice Cream? Yes!…

F

Functional Design in Clojure

1
Ep 093: Waffle Cakes 14:22

۲ years پیش14:22

14:22

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "effective composition." We search for that sweet spot between full-featured mixes and simple ingredients when crafting your software recipes. Our discussion includes: What makes code "composable"? What makes it "not composable"? What baking has in common with software design. What baking does not have in common with software design. Are you using ingredients or a box mix? How to know when you need to split something apart. When is a part too big? When is a part too small? What is too flexible? The principle of orthogonality. Decomposing without the smell. Selected quotes: "This is a Clojure podcast. We have to have a definition every once in a while." "When you try to take code that someone has made and combine it with other things, how much resistance, how much difficulty, do you begin to encounter?" "A cake is a composition of smaller things." "You're still composing a cake. You're just composing it out of bigger things." "Let's compose dinner together." "You might be fighting against some of the ingredients, because you have a whole lot of stuff that's been premixed that comes along for the ride." "What if you want waffles?" "Everything is made of atoms! If I could assemble atoms, I have ultimate flexibility!" "If it's narrowly focused, it's hard for it to get in the way of other things." "You're making well-suited things, that have small scopes of responsibility, so that you can weave them together appropriately for your domain." "Who owns the recipe?" "Ah yes, you are the cause and the solution of the problem!" "Are you able to write a new recipe and reuse all the ingredients?" Links: Ep 033: Cake or Ice Cream? Yes! Ep 065: Stuck in the Middle Ep 079: Compose Thyself…

F

Functional Design in Clojure

1
Ep 092: Freeing Limits 17:02

۲ years پیش17:02

17:02

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "freedom through constraints." We bump into limiting constraints and learn to love their freedoms. Our discussion includes: Why expressiveness isn't everything. Causes of combinatorial complexity. The long-term benefits of decreased complexity. Problems Clojure eliminates with its language design. Why complexity feels so good when you're problem solving. How the right limits create freedom. Why Clojure's limiting constraints make it hard to switch from object-oriented programming. Reasons software teams start using microservices. Integrating around behavior vs integrating around data. Awesome hair. Selected quotes: "Complexity really adds up the more you load it into your application." "Clojure eliminates the time sink related to combinatorial complexity while giving you the ability to craft things faster." "When you're inside a method of an object, there's all those delicious member variables, instance variables, that are just sitting there ready to be touched--ready to be modified--and you have to be super disciplined not do that." "You feel so good because of all the systems you come up with, and all the structure, and all the rules. You notice patterns in the different rules, and you publish books about them, and you think, 'Wow! Look how productive we've been in coming up with all these ways of managing all this complexity! Programming is so hard!'" "Limiting constraints create freedom through limitation." "Sorry, I mutated it out from underneath us." "Eventually, over time, [the OO system] grows and then it becomes unmanageable. It becomes ungovernable. There are no rules that you can actually have, so the only solution is to rewrite." "You know it's a good metaphor when you can take it to the level where it starts falling apart." "This is a big downside of Clojure: you don't just pick it up like your last four programming languages that were all from imperative land." "It's not just a functional style that hangs out as a little part of your program. This is functional: top to bottom, ceiling to floor." "You think of zucchini bread as a naturally mom-shaped problem." "You have to discover another way to do it, and that can take time." "It appeared again! Great! It's for me!" "I just want my program to work, and some guy, with awesome hair, has told me that this is better, but I don't know how to get there!" Links: 056: Opt-In Complexity 091: Combo Boost…

F

Functional Design in Clojure

1
Ep 091: Combo Boost 12:00

۲ years پیش12:00

12:00

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "effective expressiveness." We compose our thoughts on why Clojure expressiveness is so effective but can be so hard to learn. Our discussion includes: Why Clojure can give you a boost as a developer—in the short and long term. Our definition of expressiveness, and why it's not simply about conciseness. Why object-oriented developers often struggle to learn functional programming. What makes some abstractions better than others. The deep implications of immutability. How to work your way up to an expert functional programmer. Selected quotes: "Getting rid of problems over time really adds up." "More of the code is solving the problem instead of being boilerplate." "You hold things at a higher level, but still very clearly, because they're well defined." "Because all of the verbs in Clojure work on all the data structures, they become more powerful." All of these new pieces have names and concepts associated with them, and you're not going to know them." "You can do tiny things, but tiny things are toys. You want big things. You want to solve problems." "It's not a small adaptation of thinking in a little region of the code, but it affects how you structure and organize everything." "You have to go in levels. You can't go all the way from zero to done." Links: Clojure Core…

F

Functional Design in Clojure

1
Ep 090: Conditional Results 25:00

۵ years پیش25:00

25:00

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "cond-> and cond->>." We devote some time to two functions that are indispensable when computations require variation. Selected quotes: "Our love of Clojure is unconditional!" "These are all about conditional operations." "It's shorter and easier to understand, which is uncommon in development." "That gave me data I could reason about in the program." Links: cond-> cond->>…

F

Functional Design in Clojure

1
Ep 089: Branching Out 38:47

۵ years پیش38:47

38:47

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "if, when, case, condp, and cond." We wander through the myriad ways of making decisions and listing choices in Clojure. Selected quotes: "Clojure is all about the practical." "Never be afraid of the in-line if." "We need to conserve parentheses, for future generations of lispers." "Build up the language to the vocabulary of your domain, and you won't have to think about the language any more, you'll just be thinking about your problem." "Where are you in programming, without branching?" Links: if when case cond condp Related Episodes: 086: Let Tricks 087: Polymorphic Metal…

F

Functional Design in Clojure

1
Ep 088: Do Dorun Run! 26:40

۵ years پیش26:40

26:40

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "doall, dorun, doseq, and run!." We eagerly discuss the times we need to interact with the messy world from our nice clean language. Selected quotes: "We're going to talk about not being lazy, for once." "Laziness is a virtue." "Clojure loves immutability. The world loves mutability." Links: doall dorun doseq run! Larry Wall's Three Virtues Related Episodes: 014: Fiddle with the REPL 085: For for the When 086: Let Tricks…

F

Functional Design in Clojure

1
Ep 087: Polymorphic Metal 29:55

۵ years پیش29:55

29:55

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "multimethods." We discuss polymorphism and how we tackle dynamic data with families of functions. Selected quotes: "You don't have to reach for polymorphism as quickly in Clojure." "Polymorphism at its simplest." "A natural fit for processing a list of heterogeneous things." "Our natural tool of computation: the function." Links: defmulti defmethod Multimethods and Hierarchies defmethod in Pyrite font…

F

Functional Design in Clojure

1
Ep 086: Let Tricks 32:49

۵ years پیش32:49

32:49

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "let." Let us share some tricks to reduce nesting and make your code easier to understand. Selected quotes: "It's really about avoiding nesting." "Functions grow as a function of their nesting." "I don't want to start doing math on a nil!" "It's a clue that the value might not be there." "It's like your prep space in the kitchen." Links: let…

F

Functional Design in Clojure

1
Ep 085: For for the When 29:27

۵ years پیش29:27

29:27

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "for." We talk about this data generating macro, while we remember situations when it was useful. Selected quotes: "You are generating a new sequence in a declarative way." "In typical Clojure fashion, I looked up the definition." "Forgive us for our puns." "Where the while is in the for. Another statement that doesn't make sense to the outside listener." Links: for…

F

Functional Design in Clojure

1
Ep 084: All Sorts 27:07

۵ years پیش27:07

27:07

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "sort and sort-by." We lay out a list of ways to sort your data, ordered by their relative power. Selected quotes: "Clojure core is a shared vocabulary that we all can use." "I can't sort things in my mind, I'll just have the computer do it for me." "There only needs to be one sort-by function, because of the flexibility of passing in the classifying function." "We don't need any framework-y mumbo-jumbo." "We do interop, but only at the edges." Links: sort sort-by…

F

Functional Design in Clojure

1
Ep 083: Sift Sans Sort 24:26

۵ years پیش24:26

24:26

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "partition-by and group-by." We get a handle on big buckets of data by sifting elements into smaller buckets. Selected quotes: "If you know the core toolbox better, you will use it more effectively." "Is there a group-by hiding in this code that uses filter?" "Be eager to be lazy." Links: partition-by group-by…

F

Functional Design in Clojure

1
Ep 082: Filter and Friends 27:55

۵ years پیش27:55

27:55

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "filter, filterv, remove, keep, and keep-indexed." We talk about sifting data and marvel at the simple function that can turn two steps into one. Selected quotes: "It's not just linguistic, it helps it make more sense." "We should be more positive about our intent." "Clojure's sense of falsey is thankfully limited to nil and false." "Nil is a positive statement that it didn't work out." "Keep is great for those times where you need data for both the transform and the filter." "It's only a complement away!" "Reaching for optimization too early leads to more complexity." Links: filter filterv remove keep keep-indexed…

F

Functional Design in Clojure

1
Ep 081: List No More 26:05

۵ years پیش26:05

26:05

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "first, second, key, and val." We talk about positive nothing and the proliferation of tuples. Selected quotes: "Let our ramblings help your code ramblings." "First is the key." "Nil, the positive statement of nothing." "Key and val are a hint that the bit of data you're working on is really a small part of a map." Links: first second key val…

F

Functional Design in Clojure

1
Ep 080: Apply as Needed 22:14

۵ years پیش22:14

22:14

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "apply." We take time to unroll some examples of this function. Selected quotes: "If you go and read the code in core, which I would encourage." "And all the rest..." "Why is there no third function?" "If you're using reduce, you may be missing an opportunity to use apply." Links: apply…

F

Functional Design in Clojure

1
Ep 079: Compose Thyself 27:40

۵ years پیش27:40

27:40

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "comp." We create a whole episode by combining examples of useful uses of comp. Selected quotes: "It's the Voltron of functions." "Our only tool for the job is functions." "Getting your higher-order functional thinking going." "Not only is it concise, it's really easy to understand, which is one of the awesome superpowers of Clojure." Links: comp Related Episodes: 061: Transcendental Transformations 076: Multiple Views on Juxt 078: Impartial Thoughts…

F

Functional Design in Clojure

1
Ep 078: Impartial Thoughts 25:33

۵ years پیش25:33

25:33

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "partial." We cover some of the ways we use partial, without getting too literal. Selected quotes: "We love punning so much, we pun in our code." "It didn't help that I encountered this in academia." "Not the function you have, but the function you need." "I would use partial more if it wasn't so long." Links: partial Episodes: 014: Fiddle with the REPL…

F

Functional Design in Clojure

1
Ep 077: Thread Some 23:10

۵ years پیش23:10

23:10

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "some-> and some->>." We spend some time going through how these macros help keep our code nil-safe. Selected quotes: "Keep trying till you hit a nil and then stop." "That seemed all well and good, and then I hit nil." "It ends up being just a bit more concise." "A nice way of testing something and then being able to use it right away." Links: some-> some->>…

F

Functional Design in Clojure

1
Ep 076: Multiple Views on Juxt 30:08

۵ years پیش30:08

30:08

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "juxt." We take a turn with juxt, looking at all the ways it can help line up data. Selected quotes: "When it feels like core is missing something obvious, there's probably an idiom you need to learn." "The way you become an excellent writer of Clojure is by reading excellent Clojure code." Links: juxt Advent 2019 part 4, A useful idiom Bobby Towers' "Just Juxt" posts Episodes: 012: Embrace the REPL 013: Connect the REPL 014: Fiddle with the REPL…

F

Functional Design in Clojure

1
Ep 075: Merge With Fun 26:08

۵ years پیش26:08

26:08

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "merge-with." We focus in on merge-with, a powerful function for aggregating data. Selected quotes: "We pick up idioms from other people." "What sounds insane at face value actually starts to make a lot of sense." "It's just a filter, you don't even have to write a function for that." "The important part in the middle, where we might mess up, is all pure." "We don't just have an O(n) problem, we have an 0(n) API calls problem." Links: merge-with Episodes: 047: What is "nil punning"?…

F

Functional Design in Clojure

1
Ep 074: Deploy, They Said. It'll Be Fun, They Said. 27:35

۵ years پیش27:35

27:35

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "Deploying Clojure." We survey the myriad ways we've used to launch our code into production, and laugh about the various complexities we've found. Selected quotes: "How do you get closure on the development of your Clojure code?" "What do you mean by production?" "Tomcat goes to war." "That was back before containers." "Trade a little bit of money for a lot of sanity." "The problem with these things is the configuration you end up with is really simple after you've read 8 hours of documentation." "The number of lines of code does not indicate the amount of effort." "Every time you accept a new component into your system, you are also accepting its complexity."…

F

Functional Design in Clojure

1
Ep 073: Silent Sockets 38:19

۵ years پیش38:19

38:19

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "Checking websocket health." We worry about the health of our websockets and, after looking for help from the standards bodies, roll up our sleeves and handle it ourselves. Selected quotes: "It's the ghosting in the websocket world." "With polling, every request is a fresh opportunity to recover from the last request's errors." "Long polling is just beating the server up at a slower rate." "By giving your code names with meaning, you can read the code that uses it and comprehend it faster." "Complexity costs, and you pay in time and confusion." Related episodes: 012: Embrace the REPL 020: Data Dessert…

F

Functional Design in Clojure

1
Ep 072: Sea of Sockets 32:11

۵ years پیش32:11

32:11

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "Organizing our websocket code." We switch to using a component to manage our websockets, enabling ease of development and future growth. Selected quotes: "It feels like every time I use a global, a butterfly dies." "We don't want to leave the REPL!" "It's important to keep the abstraction un-leaky." "When you lower the cognitive load of programming, that makes it more enjoyable."…

F

Functional Design in Clojure

1
Ep 071: Socket Synergy 29:10

۵ years پیش29:10

29:10

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "Using websockets for notification." We wander into the weeds, jumping through the myriad hoops required to deliver a simple notification. Selected quotes: "It's like we're taking a regular series and upgrading it somehow." "Welcome to programming, where you'll spend hours diving into rabbit holes; and still love your work!" "That sentence you just said sounds something they would say in a movie that would make all the programmers in the audience roll their eyes simultaneously." "We just replaced one problem with another." Related episodes: 053: How can I save my data from serialization?…

F

Functional Design in Clojure

1
Ep 070: Socket Circus 29:43

۵ years پیش29:43

29:43

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "Websockets." We talk about spontaneously sending data from the server to the client to address our users' insecurities. Selected quotes: "Classic engineering solution, let's just bend a couple of assumptions." "Reloading the web page is 1995-style integration." "Poll every tenth of a second? Yeah, that'll work till we need to scale past 4 users." "In every engineering choice, there is always a tradeoff."…

F

Functional Design in Clojure

1
Ep 069: Stuck in the Web 28:50

۵ years پیش28:50

28:50

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "Request handler pitfalls." We examine our history writing web handlers and talk about all the ways we've broken them. Selected quotes: "Doing a pitfalls episode is easy, because you just need to look at your scars." "Letting exceptions escape is TMI for the web." "Multiple error handling middleware is ok. They're in place, just in case." "Smaller doesn't necessarily mean simpler." "A little bit of repetition can lead to a lot of clarity." "Java can turn just about anything into a string."…

F

Functional Design in Clojure

1
Ep 068: Static on the Line 24:20

۵ years پیش24:20

24:20

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "Serving static assets." We tease apart the layers involved in serving static assets and are surprised by how many we find. Selected quotes: "You'd think that serving up plain old files would be nice and simple." "MIME types are an explosion of complexity." "Simplicity is the only remedy for complexity. Abstraction is not a remedy for complexity."…

F

Functional Design in Clojure

1
Ep 067: Handling Handler's Handles 23:52

۵ years پیش23:52

23:52

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "Resource management for web handlers." We manage to find a way to get our handlers the resources they need to get real work done. Selected quotes: "Let's get some work done in these web handlers!" "The handler needs a handle." "Handlers do not live on a desert island, alone." Links: Component Integrant Mount Related episodes: 012: Embrace the REPL 013: Connect the REPL 014: Fiddle with the REPL…

F

Functional Design in Clojure

1
Ep 066: Compose the Codec 28:05

۵ years پیش28:05

28:05

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "Making middleware to handle JSON." We reinvent the wheel, and along the way discover a few reasons why you might want to do so as well. Selected quotes: "Ring's choice: all of HTTP is a function." "Every concern is a function you compose in." "It's like a bucket brigade with the request." "Solving JSON encoding and decoding is a cross-cutting concern." Links: ring-json - Ring middleware for handling JSON…

F

Functional Design in Clojure

1
Ep 065: Stuck in the Middle 30:41

۵ years پیش30:41

30:41

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "Ring middleware." We find that the middle is a very good place to start when almost everything is composed functions. Selected quotes: "It's all fun from Ring on up!" "I want to have more than one handler." "We want to build up different scenarios declaratively." "The way you compose functions together is by building a function that takes functions and returns functions." Links: Ring Wiki Ring concepts Ring examples Third party libraries Parameters middleware Routing middleware examples compojure reitit…

F

Functional Design in Clojure

1
Ep 064: Put Ring on It 19:45

۵ years پیش19:45

19:45

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "Ring, the foundation of Clojure HTTP" We focus on the bedrock abstraction for all Clojure web applications and marvel at its simplicity. Selected quotes: "Somebody wants to make a web app, and suddenly they're up against all these layers." "The law of leaky abstractions is good to remember." "HTTP is just a function call over the Internet." "Your function gets a map, and it returns a map."…

F

Functional Design in Clojure

1
Ep 063: Web of Complexity 22:38

۶ years پیش22:38

22:38

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "HTTP and the Web of Complexity." We launch a new series and immediately get tangled in the many layers that make up the web. Selected quotes: "Clojure has several ways of getting to the web." "There really are a lot of layers." "There are a lot of bad requests out there." "For all the decodings, there is an encoding."…

F

Functional Design in Clojure

1
Ep 062: 2019 in Review 36:03

۶ years پیش36:03

36:03

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "2019." We look back at the past year and highlight our favorite episodes. Selected quotes: "2020 is the year of perfect hindsight." "Even we are surprised at what we recorded." "Mutation is like weak bricks in your solution tower, they could fail you at any time." "The Twitter API is just a whole bag of side effects." "Programming without I/O is just warming your room." "Not everything does all the things perfectly in all the ways." "Complexity really is the killer in long term projects." Related episodes: The REPL Series (12-14) 012: Embrace the REPL 014: Fiddle with the REPL The Time Tracking Series (15-20) 016: When 8 - 1 = 6 018: Did I Work Late on Tuesday? The Twitter Poster Series (21-27) 022: Evidence of Attempted Posting 024: You Are Here, But Why? The Donut Logs Series (28-35) 032: Call Me Lazy 034: Break the Mold Advocacy Episodes (36-39) 036: Why Do You Recommend Clojure? 037: What Advice Would You Give to Someone Getting Started With Clojure? 038: How do I convince my coworkers to use Clojure? 039: Why use Clojure over another functional language? 040: Should I use lein, boot, or tools.deps? Data Episodes (41-42) 041: Why do Clojurians make such a big deal about immutability? 042: What does it mean to be 'data-oriented'? Questions from our listeners (43-48) 046: When is Clojure not the right tool for the job? 047: What is "nil punning"? 048: Help! How do I fix my REPL? Word! Word! Word! Series (aka Nountober) (49-52) 049: Keywords! Keywords! Keywords! 052: Functions! Functions! Functions! 053: How can I save my data from serialization? 054: The Forest in the Trees 055: Sets! What are they good for? 056: Opt-In Complexity 057: Clojure/Conj 2019 Recap Reduce, Reducers, and Transducers (58-61) 061: Transcendental Transformations…

F

Functional Design in Clojure

1
Ep 061: Transcendental Transformations 28:50

۶ years پیش28:50

28:50

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "Transducers." We unpack transducers and find a familiar pattern that enables freedom from collections. Selected quotes: "The humble sequence. It is a mighty abstraction." "Your reducing function can get more abstract by passing in the reducing function that handles producing the result." "This is mind-bending enough, it's worth saying again in a different way." "We have a function that returns a function that returns a function!" "The cool thing about reducing functions is that they're not about collections. They are about reduction." Links: Transducers - Reference Transducers are Coming - Rich's introductory blog post What are good use cases for transducers? - From the Clojure FAQ Reducers, transducers and core.async in Clojure…

F

Functional Design in Clojure

1
Ep 060: Reduce Done Quick 26:40

۶ years پیش26:40

26:40

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "Reducers." We look at clojure.core.reducers and how it extracts performance by composing reducing functions. Selected quotes: "The seq abstraction, it's rather lazy." "It's amazing how much work laziness requires." "It doesn't work out as well. That's a wonderful way of saying 'wrong'." "Because ease of use was a design goal, they're pretty easy to use." Links: Reducers - Reference Reducers - A Library and Model for Collection Processing - Rich's introductory blog post Know Your Reducers Reducers, transducers and core.async in Clojure…

F

Functional Design in Clojure

1
Ep 059: Lining It Up 32:06

۶ years پیش32:06

32:06

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "Sequences." We examine the sequence abstraction and then ponder how it helps and hinders our data transformation. Selected quotes: "There is only one filter, map, and reduce. They need to be able to interpret all collections." "Primitives only take you so far." "The first thing is the first thing!" "The seq abstraction is foundational to Clojure core." "The into function pours a sequence back into a concrete data structure."…

F

Functional Design in Clojure

1
Ep 058: Reducing It Down 28:52

۶ years پیش28:52

28:52

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "Reduce and reducing functions" We take a long hard look at reduce and find the first of many generally useful nuggets inside. Selected quotes: "Parentheses are a joy, they are hugs around your code." "You want to boil it down into a result, so you use reduce." "We've taken ahold of the cooking metaphors and are just running with it!" "Reducing functions are a backbone of functional programming, because we don't have mutation." "Understanding reduce is really important for understanding functional programming." Related episodes: 002: Tic-Tac-Toe, State in a Row…

F

Functional Design in Clojure

1
Ep 057: Clojure/Conj 2019 Recap 34:51

۶ years پیش34:51

34:51

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "Clojure/Conj 2019 Recap" We go through our notes and recall the most memorable talks from the Conj last week. Selected quotes: "Cold climate, warm people." "Learn skills that transfer." "The future is about immutability and composability." "YAML! YAML! YAML! The names of an episode we have not done, and probably will not do." "If you can make solving problems easier by using Clojure, then there is a lot of incentive to learn and use Clojure." "You can't get rid of bias, even when you know you have it. All you can do is change how you work." "The Conj has become less about Clojure and more about solving problems." Talks: Defeating the Four Horsemen of the Coding Apocalypse - Mark Bastian Follow the Data: Product Development in Clojure - Derek Troy-West Application: Operatr Love Letter To Clojure: And A Datomic Experience Report - Gene Kim New book: The Unicorn Project Goodbye YAML: Infrastructure as Code in Clojure - Eno Compton & Tyler van Hensbergen Interactive Programming for Artificial Intelligence - Dragan Djuric Libraries: uncomplicate.org , Blog: dragan.rocks , Books: aiprobook.com Probabilistic Record Linkage of Hospital Patients - Chris Oakman Rapid Prototyping for Software Development - Sara Kimmich (Architecture) Diagrams as Data - Avi Flax The Hitchhikers Guide to Multiparadigm Programming - Ariel Ortiz A Racket Perspective on Research, Education, and Production - Matthew Flatt Site: racket-lang.org Composable Tools - Alex Miller Sherlock Holmes, Consulting Developer - Stuart Halloway…

F

Functional Design in Clojure

1
Ep 056: Opt-in Complexity 28:25

۶ years پیش28:25

28:25

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "Opt-In Complexity" We discuss complexity and try to come up with a simple explanation for why Clojurians avoid it so ruthlessly. Selected quotes: "Abstraction: the thing that creates more complexity while making you feel like you have less." "Clojure people seem to be allergic to complexity." "Frameworks are the breeding ground for complexity." "Complexity is the enemy of long lived code bases." "Each part of the application is in its own vault of isolation, and they only get to come together and interact in the lobby." "You can't be afraid of something that hasn't scared you yet." "It's very difficult to say no once you've said yes." Links: Simple Made Easy Luminus Hoplon Pedestal…

F

Functional Design in Clojure

1
Ep 055: Sets! What Are They Good For? 28:25

۶ years پیش28:25

28:25

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "Sets! What are they good for?" We examine one of the lesser used data structures in Clojure and talk about its unique characteristics and uses. Selected quotes: "Sets aren't going to get top billing." "Sets are really about uniqueness." "Sets let us calculate differences cheaply." "Clever is a word I used to like when I was a younger programmer." "Clever now equals hours of suffering!"…

F

Functional Design in Clojure

1
Ep 054: The Forest in the Trees 21:58

۶ years پیش21:58

21:58

Each week, we discuss a different topic about Clojure and functional programming. If you have a question or topic you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the topic is: "Working with heavily nested trees." We discuss three powerful libraries (Specter, Spectacles, and clojure.walk) and where they might fit into our Clojure programs. Selected quotes: "Tree manipulation is like surgery, you need to get down to the right level before you start making changes." "I've seen people come up with novel and interesting solutions to problems that other people just don't have." "Spec is a structural type checker, instead of a name-based type checker, which is far more useful." "Every time you reach for a tool, you're inviting its complexity into your project." "The structure of your data will lead you toward a particular tool." Related episodes: 020: Data Dessert - Summary episode for the time log series Links: clojure.walk Specter Spectacles…

F

Functional Design in Clojure

1
Ep 053: How Can I Save My Data From Serialization? 30:36

۶ years پیش30:36

30:36

Each week, we answer a different question about Clojure and functional programming. If you have a question you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the question is: "How can I save my data from serialization?" We record our thoughts on the many trade-offs we have encountered preserving our data when it leaves our programs. Selected quotes: "Clojure has the best data structures on the planet." "One of the reasons why JSON is so popular is because XML was so overbearingly hard." "XML decided that trees aren't dimensional enough, so they added another dimension in attributes." "Serialization formats aren't for humans, Nate, what are you talking about?!" "EDN isn't horrible to look at because it's just Clojure data structures." "We don't want freezer burn!" Related episodes: 014: Fiddle with the REPL Links: EDN Transit Nippy Fressian fress: Fressian for Clojure(Script) AND WASM!…

F

Functional Design in Clojure

1
Ep 052: Functions! Functions! Functions! 37:27

۶ years پیش37:27

37:27

Each week, we discuss a different topic about Clojure and functional programming. If you have a question you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, our topic is: "Functions! Functions! Functions!" We wonder how we could function without these critical building blocks, so we catagorize their varied uses. Selected quotes: "Functions put the fun in functional." "Each function category has a different mindset." "Programming is more fun when you're only thinking about the problem in front of you." "Naming bits of computation helps you understand an abstraction without having to dip into it." "So you don't want the function to be called 'write-or-fetch-or-sleep-or-return!'?" "Side effecting functions shouldn't think. They are grunts." "The I/O fairy hands you data, and you get it all transformed with pure functions, and then you hand it back to the I/O fairy." "That Twitter use case, that has a lot more side effects." Related episodes: 002: Tic-Tac-Toe, State in a Row 024: You Are Here, But Why?…

F

Functional Design in Clojure

1
Ep 051: Maps! Maps! Maps! 28:01

۶ years پیش28:01

28:01

Each week, we discuss a different topic about Clojure and functional programming. If you have a question you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, our topic is: "Maps! Maps! Maps!" We discuss maps and their useful features, including a key distinction that we couldn't live without. Selected quotes: "Working with Clojure makes you feel like you're living in the future!" "Maps are bags of dimensions." "The namespace of the key is the entity, the name is the attribute, and the value is the value." "Flatter maps make it so you end up writing less code." Related episodes: Time Log series: 015: Finding the Time 016: When 8 - 1 = 6 017: Data, at Your Service 018: Did I Work Late on Tuesday? 019: Dazed by Weak Weeks 020: Data Dessert 049: Keywords! Keywords! Keywords! Links: Love Letter To Clojure (Part 1) - Gene Kim Code sample: ;; Player records: one nested, one with rich keys. ( def players-nested [{ :player { :id 123 :name "Russell" :position :point-guard } :team { :id 432 :name "Durham Denizens" :division :eastern }} { :player { :id 124 :name "Frank" :position :midfield } :team { :id 432 :name "Durham Denizens" :division :eastern }}]) ( def players-rich [{ :player/id 123 :player/name "Russell" :player/position :point-guard :team/id 432 :team/name "Durham Denizens" :team/division :eastern } { :player/id 124 :player/name "Frank" :player/position :midfield :team/id 432 :team/name "Durham Denizens" :team/division :eastern }]) ;; Extract player and team id, along with team name ; Nested ( defn extract [ player ] ( let [{ :keys [ player team ]} player ] { :player ( select-keys player [ :id ]) :team ( select-keys team [ :id :name ])})) # _ ( map extract players-nested ) ; ({:player {:id 123}, :team {:id 432, :name "Durham Denizens"}} ; {:player {:id 124}, :team {:id 432, :name "Durham Denizens"}}) ; Rich # _ ( map #( select-keys % [ :player/id :team/id :team/name ]) players-rich ) ; ({:player/id 123, :team/id 432, :team/name "Durham Denizens"} ; {:player/id 124, :team/id 432, :team/name "Durham Denizens"}) ;; Sort by team name and then player name ; Nested # _ ( sort-by ( juxt #( -> % :team :name ) #( -> % :player :name )) players-nested ) ; ({:player {:id 124, :name "Frank", :position :midfield}, ; :team {:id 432, :name "Durham Denizens", :division :eastern}} ; {:player {:id 123, :name "Russell", :position :point-guard}, ; :team {:id 432, :name "Durham Denizens", :division :eastern}}) ; Rich # _ ( sort-by ( juxt :team/name :player/name ) players-rich ) ; ({:player/id 124, ; :player/name "Frank", ; :player/position :midfield, ; :team/id 432, ; :team/name "Durham Denizens", ; :team/division :eastern} ; {:player/id 123, ; :player/name "Russell", ; :player/position :point-guard, ; :team/id 432, ; :team/name "Durham Denizens", ; :team/division :eastern})…

F

Functional Design in Clojure

1
Ep 050: Parentheses! Parentheses! Parentheses! 27:17

۶ years پیش27:17

27:17

Each week, we discuss a different topic about Clojure and functional programming. If you have a question you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, our topic is: "Parentheses! Parentheses! Parentheses!" We defend the lowly parentheses, and discuss the benefits of having this stalwart shepherd dutifully organizing our code. Selected quotes: "I'm converting my visual cues into audio information." "Parentheses are like dust bunnies that have stacked up over time." "Pounds of parentheses." "The parenthesis is greatly misunderstood." "Parentheses make the grammar of the language incredibly simple." "I've never seen that in math, at least not in normal math." "If you have 15 close parentheses, you might have to do some refactoring." "Parentheses are the hugs your program needs." "Peace, Love, Parentheses."…

F

Functional Design in Clojure

1
Ep 049: Keywords! Keywords! Keywords! 25:25

۶ years پیش25:25

25:25

Each week, we discuss a different topic about Clojure and functional programming. If you have a question you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, our topic is: "Keywords! Keywords! Keywords!" We examine all the fascinating properties of keywords, how to use them, and why they're so much better than strings and enums. Selected quotes: "The question of the day is: Why is there no question of the day?" "Everything behind the scenes should get turn into integers because that is the good and true way." "A FAQ after the fact." "We forget about identical? because Clojure gives us value semantics!" "Magic numbers you can spell with letters." "The ability of communicating with letters--something we humans have been developing for a while." "What is this sorcery?!" "JSON. That's not not a guy who likes strings." "Let's blame JSON." "If you don't have to recursively walk a tree, why recursively walk a tree?" "A little bit of copying is better than a lot of depending." "The randomly placed last one wins!" "Trust but verify."…

F

Functional Design in Clojure

1
Ep 048: Help! How Do I Fix My REPL? 27:03

۶ years پیش27:03

27:03

Each week, we answer a different question about Clojure and functional programming. If you have a question you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the question is: "Help! How do I fix my REPL?" We catalog the many ways we've broken our REPLs and talk through our strategies for getting back on track. Selected quotes: "The REPL is a wonderful, wonderful thing." "A cleansing restart." "What is my Clojure band name?" "tools.namespace will help you find rename errors early." "Refresh will continue to work, as long as you continue to work." "Even a defonce can be def'd again. It's a defonce, for now."…

F

Functional Design in Clojure

1
Ep 047: What Is "Nil Punning"? 28:09

۶ years پیش28:09

28:09

Each week, we answer a different question about Clojure and functional programming. If you have a question you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the question is: "What is 'nil punning'?" We gaze into the nil and find a surprising number of things to talk about. Selected quotes: "The lowly, magnificent nil. Some people love it, some people hate it." "Null is the value you give your program if you want to see it die." "Nil is not null." "This function found nothing, and I passed that to the next function, and it found nothing in the nothing." "It's amazing how much nothing you can find in nothing." "You can pull data out without fear." "What does a nil Cat look like?" "A lot of arithmetic stuff is nil-intolerant." "No answer isn't going to start becoming an answer later."…

F

Functional Design in Clojure

1
Ep 046: When Is Clojure Not the Right Tool for the Job? 25:06

۶ years پیش25:06

25:06

Each week, we answer a different question about Clojure and functional programming. If you have a question you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the question is: "When is Clojure not the right tool for the job?" We look at the varied forms that Clojure can assume and consider where it might not fit. Selected quotes: "Just reject all the jobs where Clojure is inappropriate and you'll always be able to use Clojure." "The JVM is like a super tanker. It will get a whole lot of work done. But it takes a little while to get going." "On the language level, there are social aspects. Bringing in Clojure covertly will make the rest of the team angry." "A lot of times, when talking about technical things, we tend to overlook the human effects." "Two questions: Is the technology suitable? Are we going to get screwed if we go down this path?" "There is a conceptual leap into functional programming that is non-trivial." "People will have a hard time letting go of their existing language until they can feel how Clojure is different." Related episodes 038: How do I convince my coworkers to use Clojure?…

F

Functional Design in Clojure

1
Ep 045: Why Have Derived Fields in Data When I Can Just Calculate Derived Data as Needed With a Function? 20:57

۶ years پیش20:57

20:57

Each week, we answer a different question about Clojure and functional programming. If you have a question you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the question is: "Why have derived fields in data when I can just calculate derived data as needed with a function?" We take a focused look at the balance of using functions or derived fields and where each is preferable. Selected quotes: "Clojure has a gravity toward using the built-in data structures." "Immutability helps you handle concurrent updates to dependent fields without using locks." "If you add derived fields, you can query the data using information at different levels of abstraction."…

F

Functional Design in Clojure

1
Ep 044: What's So Different About Clojure's REPL? 28:15

۶ years پیش28:15

28:15

Each week, we answer a different question about Clojure and functional programming. If you have a question you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the question is: "What's so different about Clojure's REPL?" We evaluate what a REPL really is and show that it's much more about the developer experience than simply calculating values. Selected quotes: "The Clojure REPL is your window into the state of a running application." "It's a much larger, broader kind of experience." "It's funny to have mutable programs and immutable data, but that's what Clojure gives you." "The fact that the REPL can read expressions and run them becomes one of it's least interesting attributes." Related episodes 012: Embrace the REPL 013: Connect the REPL 014: Fiddle with the REPL…

F

Functional Design in Clojure

1
Ep 043: What Is 'Faking' a Resource? 23:54

۶ years پیش23:54

23:54

Each week, we answer a different question about Clojure and functional programming. If you have a question you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the question is: "What is 'faking' a resource?" We talk about the virtues of faking and then outline several real techniques for getting work done. Selected quotes: "We like to mock mocking, so don't mock us if we mock mocking." "No object is an island, they always refer to each other." "It's all about getting work done." "Why make my own fake data when the computer is so good at it?" "It's like training a small diminutive version of the external service to behave in the ways you want it to so you can get stuff done." "Fake it while you make it." Related episodes 014: Fiddle with the REPL 025: Fake Results, Real Speed…

F

Functional Design in Clojure

1
Ep 042: What Does It Mean to Be 'Data-Oriented'? 26:47

۶ years پیش26:47

26:47

Each week, we answer a different question about Clojure and functional programming. If you have a question you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the question is: "What does it mean to be 'data-oriented'?" We merge together different aspects of Clojure's data orientation, and specify which of those help make development more pleasant. Selected quotes: "Clojure has the corner on data." "Other languages have data too, it's just locked in little cages." "Data is inert, it can't harm you." "Because Clojure is expressed in its own data structures, and those structures are simple, that makes Clojure syntax simple." "Find a good way to represent the information that you want to work with, in a way that feels appropriate for the subject matter." "If you find a good way of representing your information, that representation tends to be pretty stable. All of the change is in the functions you use to work with it."…

F

Functional Design in Clojure

1
Ep 041: Why Do Clojurians Make Such a Big Deal About Immutability? 27:11

۶ years پیش27:11

27:11

Each week, we answer a different question about Clojure and functional programming. If you have a question you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the question is: "Why do Clojurians make such a big deal about immutability?" We cover several practical side effects of immutability and why we've become such big fans of data that doesn't let us down. Selected quotes: "Well, we don't have Monads to talk about." "What good is a program, if you can't change stuff!?" "The memory cost of data structures is in proportion to the changes, not the users." "I can hang on to a reference to the old state and a reference to the new state very cheaply." "You can reduce comparison to referential equality." "Once you can efficiently save every version of the state, going back to a previous version is no big deal." "I love determinism. Determinism is trustable." "I can trust immutable data. And if I can trust it, then it can occupy a smaller part of my brain." Related episodes: 002: Tic-Tac-Toe, State in a Row 039: Why use Clojure over another functional language?…

F

Functional Design in Clojure

1
Ep 040: Should I Use Lein, Boot, or Tools.deps? 28:04

۶ years پیش28:04

28:04

Each week, we answer a different question about Clojure and functional programming. If you have a question you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the question is: "Should I use lein, boot, or tools.deps?" We assemble a list of build tool characteristics and talk about how each tool stacks up before giving our recommendations. Selected quotes: "The best thing about boot is that you can use all the flexibility of Clojure code. The worst thing about boot is that you can use all the flexibility of Clojure code." "Tools.deps, the non-build build tool." "I have a file, it's got Clojure in it, and I want to run it." "Tools.deps has enabled people to make build functions that stand alone. They don't have any ceremony to become part of the tool. It's just a main."…

F

Functional Design in Clojure

1
Ep 039: Why Use Clojure Over Another Functional Language? 25:17

۶ years پیش25:17

25:17

Each week, we answer a different question about Clojure and functional programming. If you have a question you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the question is: "Why use Clojure over another functional language?" . We examine the different categories of functional programming languages and distill out what differentiates Clojure and why we prefer it. Selected quotes: "Running just one function when developing is not only allowed in Clojure, it's encouraged and celebrated." "You don't have to make the whole world (application) agree. You can work on just a part of it and then bring it back into the rest of the world when you want it to agree." "I would like some XML in my cake." "Oh, you were a hipster Scala user." "When I pull in code off clojars, it's going to use the Clojure way, because there is a Clojure way." "If you can make all your abstractions with a simpler set of semantics, wouldn't that be better than a broader set?" "Multi-paradigm languages are inherently more complex. You really end up in the 'good parts' kind of problem. Scala, The Good Parts. Javascript, The Good Parts." "Code is about communicating with two things. The computer and the other developers. The computer can handle esoteric language features, but other developers will have a harder time with them." Related episodes: 002: Tic-Tac-Toe, State in a Row 014: Fiddle with the REPL…

F

Functional Design in Clojure

1
Ep 038: How Do I Convince My Coworkers to Use Clojure? 24:53

۶ years پیش24:53

24:53

Each week, we answer a different question about Clojure and functional programming. If you have a question you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the question is: "How do I convince my coworkers to use Clojure?" . We recall our own experiences evangelizing Clojure and give practical advice from the trenches. Selected quotes: "Don't assume someone is going to want to jump on the grenade." "There are actually two steps: 1. Convince people that Clojure is a good thing. 2. Get people to actually use it." "Lambda as in AWS Lambda, not as in Clojure lambda. They stole our cool word!" "By the way, that solution came from Clojure." "Get people thinking in the functional direction." "There are only two types of tools: those that you hate and those you don't use." "When people get stuck, they think it's the language's fault, because it has chosen to categorically eliminate solutions that they, as a developer, have relied on for years." "The path to Clojure adoption is primarily social, not technical." Related episodes: 024: You Are Here, But Why?…

F

Functional Design in Clojure

1
Ep 037: What Advice Would You Give to Someone Getting Started With Clojure? 25:57

۶ years پیش25:57

25:57

Each week, we answer a different question about Clojure and functional programming. If you have a question you'd like us to discuss, tweet @clojuredesign , send an email to feedback@clojuredesign.club , or join the #clojuredesign-podcast channel on the Clojurians Slack . This week, the question is: "What advice would you give to someone getting started with Clojure?" . We trade off giving practical tips for intrepid learners while we reminisce about our own paths into Clojure. Selected quotes: "Reading other people's code will change how you view the problem." "You're going to get stuck, but that's ok. "REPL driven development is so fast, you'll need to take a break from the whiplash." "Clojure doesn't give you breaks, like other programming languages do with their compiles and restarts." "The best code to use is code that was written by somebody else, because it's probably bug free." "The problem isn't the code that I'm writing, the problem is with my approach." "Try to solve a programming problem that you care about." Related episodes: 012: Embrace the REPL 013: Connect the REPL 014: Fiddle with the REPL…

F

Functional Design in Clojure

1
Ep 036: Why Do You Recommend Clojure? 19:58

۶ years پیش19:58

19:58

It's summertime, and that means it's time for something new. Each week, we will answer a different question about Clojure and functional programming. If you have a question you'd like us to discuss, tweet @clojuredesign or send an email to feedback@clojuredesign.club . This week, we're starting off with "Why do you recommend Clojure?" . We take turns sharing our favorite reasons, and we can't help but have fun riffing on how enjoyable Clojure is to use. Come along for the ride. Selected quotes: "Question everything else. Clojure is the answer." "Once you discover structural editing, you'll never go back." "Data is inert. It can't act." "When dealing with mutable data and side effects, it's like trying to add a domino in the middle and make sure to not break the chain." "Most of the time, running with scissors is considered to be a bad thing." "Somebody in Java has done all that plumbing that you don't really want to do."…

F

Functional Design in Clojure

1
Ep 035: Lifted Learnings 27:28

۶ years پیش27:28

27:28

Christoph and Nate lift concepts from the raw log-parsing series. Reflecting on the lessons learned in the log series. (01:15) Concept 1: We found Clojure to be useful for devops. Everything is a web application these days, "The only UIs in Devops are dashboards." For most of the series, our UI was our connected editor. We grabbed a chunk of the log file and were fiddling with the data in short order. We talk about connected editors in our REPL series, starting with Episode 12. Being able to iteratively work on the log parsing functions in our editor was key to exploring the data in the log files. (04:04) Concept 2: Taking a lazy approach is essential when working with a large data set. Lazily going through a sequence is reminiscent of database cursors. You are at some point in a stream of data. We ran into some initial downsides. When using with-open , fully lazy processing results in an I/O error, because the file has been closed already. Shouldn't be too eager too early, because then the entire dataset will reside in memory. Two kinds of functions: lazy and eager. Lazy functions only take from a sequence as they need more values. Eager functions consume the whole sequence before returning. Ensure that only the last function in the processing chain is eager. "It only takes one eager to get everybody unlazy." (08:38) Concept 3: Clojure helps you make your own lazy sequences using lazy-seq . Clojure has a deep library of functions for making and processing lazy sequences. We were able to make our own lazy sequences that could then be used with those functions. Wrap the body in lazy-seq and return either nil (to indicate the end) or a sequence created by calling cons on a real value and a recursive call to itself. (12:41) Concept 4: We work with information at different levels, and that forms an information hierarchy. The data goes from bits to characters to lines, and then we get involved. We move from lines on up to more meaningful entities. Parsed lines are maps that have richer information, and then errors are richer still. Our parsers take a sequence and emit a new sequence that is at a higher level of information. We first explored this concept in the Time series. The transformations from one level to the next are all pure. (14:53) Concept 5: Sometimes you have to go down before you can go up again another way. We pre-abstracted a little bit, and only accepted lines that had all of the data we were looking for (time, log level, etc.). Exceptions broke that abstraction, so we reworked our "parsed line" map to make the missing keys optional. (15:54) Concept 6: Maps are flexible bags of dimensions. They are a set of attributes rather than a series of rigid slots that must be filled. Functions only need to look at the parts of the map that they need. Every time we amplify the data, we add a new set of dimensions. Thanks to namespacing, all of these dimensions coexist peacefully. Multiple levels of dimensions give you more to filter/map/reduce on. Just because you distill, doesn't mean you want to lose essence. (21:09) Concept 7: Operating within a level of information is a different concern than lifting up to a higher level of information. Within a level, functions aid in filtering and aggregating. Between levels, functions recognize patterns and groupings to produce higher levels of information. Make the purpose of the function clear in how you name it. Separate functions that "lift" the data from functions that operate at the same level of information. When exploring data, you don't know where it will lead, so start by moving the data up a level in small steps. Related episodes: 012: Embrace the REPL 015: Finding the Time 028: Fail Donut 029: Problem Unknown: Log Lines 030: Lazy Does It 031: Eager Abstraction 032: Call Me Lazy 033: Cake or Ice Cream? Yes! 034: Break the Mold Clojure in this episode: lazy-seq , cons with-open…

F

Functional Design in Clojure

1
Ep 034: Break the Mold 26:44

۶ years پیش26:44

26:44

Christoph finds exceptional log lines and takes a more literal approach. Previously, we upgraded our log parsing to handle finding two errors instead of just one. "It's amazing what you don't find when you're not looking." We ended up with a set of functions that can parse out multiple errors. The result is a nice mixed sequence of errors that we can aggregate and pivot with Clojure core. "By the power of Clojure core, I will map and reduce !" (02:43) New Problem: there are exceptions in the log, and they span multiple lines! The exception continuation lines don't parse. They don't have a date, a log level, or anything else that regular log lines have. How do we collect up all those lines? They are all part of one logical error. Our error parsers don't get a chance to see these lines, because the general line parser threw them away. (06:27) Solution 1: Don't pre-parse the lines, but instead have each error parsing function do the general parse. Each parsing function would receive the raw, unparsed, line as a string. Each function would have to run parse-line on the inputs before doing their specific parsing. Now each function in our inventory needs to re-do the general parsing of the line. This ends up being much more inefficient. (10:13) Solution 2: Relax the general parsing. What is the general parser to do with a line that doesn't match the regexp? "How about if the general parse does the best job it can, and whatever it can't do, it doesn't do?" One of the keys in the data returned from parse-line is :raw/line , which is the entire line, and that's always there. So, if the regexp fails, we can always return at least that. The map doesn't have the keys that are not found. Now, parse-line will always return a map for each line. We've uplifted the data, at least slightly, from a string to a map. We can know two things Every map will have :raw/line If a map doesn't have the other keys (like :log/date ), that means it didn't parse, and is probably a continuation of a previous line. This is us expanding our program's view of reality to more closely match actual reality. We left out part of reality, and got stuck because of it. Being more literal to the data source gives us more flexibility. Clojure's dynamic nature shines in this situation. We don't have to have all the keys all the time. "A map is a bucket of dimensions." Each function that operates on the data will inspect the data and see if it has all the keys it needs. It doesn't matter if there are other keys. Namespaced keys really help in this situation, each new level of parsing can add keys without overwriting keys from other dimensions. How does this impact our parsing functions? They are unchanged, because they grab the message out using some->> , which shortcuts on nil . (19:08) Our exception error parsing function detect the start in a number of ways. The next line is a bare line? The first line ends in an opening curly brace? Then, to find the end of the exception, it can use take-while to find all the lines that are bare. To assist in finding bare lines, we can introduce a predicate function instead of embedding that logic. When all lines are found, it then combines them into a new :log/message key. When updating the error map in our parsing functions, we are careful to only operate on the data we know is there, so that other keys are not impacted. Related episodes: 028: Fail Donut 029: Problem Unknown: Log Lines 030: Lazy Does It 031: Eager Abstraction 032: Call Me Lazy 033: Cake or Ice Cream? Yes! Clojure in this episode: some->> , take-while Code sample from this episode: ( ns devops.week-06 ( :require [ clojure.string :as string ] [ devops.week-02 :refer [ process-log ]] [ devops.week-05 :refer [ parse-357-error parse-sprinkle ]] )) ( def general-re # "(\d\d\d\d-\d\d-\d\d)\s+(\d\d:\d\d:\d\d)\s+\|\s+(\S+)\s+\|\s+(\S+)\s+\|\s+(\S+)\s+\|\s(.*)" ) ( defn parse-line [ line ] ( if-let [[ whole dt tm thread-name level ns message ] ( re-matches general-re line )] { :raw/line whole :log/date dt :log/time tm :log/thread thread-name :log/level level :log/namespace ns :log/message message } { :raw/line line :log/message line })) ( defn bare-line? [ line ] ( nil? ( :log/date line ))) ( defn parse-exception-info [ lines ] ( let [ first-line ( first lines ) [ _whole classname ] ( some->> first-line :log/message ( re-matches # "([^ ]+) #error \{" ))] ( when classname ( let [ error-lines ( cons first-line ( take-while bare-line? ( rest lines ))) error-str ( string/join "\n" ( map :log/message error-lines ))] ( merge first-line { :kind :error :error/class classname :log/message error-str }))))) ( defn parse-next [ lines ] ( or ( parse-357-error lines ) ( parse-sprinkle lines ) ( parse-exception-info lines ))) ( defn parse-all [ lines ] ( lazy-seq ( when ( seq lines ) ( if-some [ found ( parse-next lines )] ( cons found ( parse-all ( rest lines ))) ( parse-all ( rest lines )))))) ( comment ( process-log "sample.log" #( ->> % ( map parse-line ) parse-all doall )) ) Log file sample: 2019-05-14 16:48:57 | process-Poster | INFO | com.donutgram.poster | transaction failed while updating user joe: code 357 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | failed to add sprinkle to donut 50493 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | sprinkle fail reason: unknown state 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | Poster #error { :cause "Failed to lock the synchronizer" :data {} :via [{:type clojure.lang.ExceptionInfo :message "Failed to lock the synchronizer" :data {} :at [process.poster$eval50560 invokeStatic "poster.clj" 40]}] :trace [[process.poster$eval50560 invokeStatic "poster.clj" 40] [clojure.lang.AFn run "AFn.java" 22] [java.lang.Thread run "Thread.java" 748]]}…

F

Functional Design in Clojure

1
Ep 033: Cake or Ice Cream? Yes! 21:22

۶ years پیش21:22

21:22

Nate needs to parse two different errors and takes some time to compose himself. Previously, we were able to parse out errors and give the parsing function the ability to search as far into the future as necessary. We did this by having the function take a sequence and return a sequence, managed by lazy-seq . (01:30) New Problem: We need to correlate two different kinds of errors. The developers looked at our list of sprinkle errors and they think that they're caused by the 357 errors. They have requested that we look at the entire log and generate a report of 357 and sprinkle errors, so we can tell if they're correlated. "When someone says, do I want cake or ice cream, the right answer is: yes, I want both!" Before, we were only parsing out a single type of error and summarizing it, but now we need to parse out both types of errors. If we try to parse both kinds of errors with the same function, we will quickly get ourselves into nested if s or maybe an infinite cond . Perhaps a complex state machine with backtracking? (05:55) Realization: Each error stands alone. Once you detect the beginning of a sprinkle error, you won't need to look for a 357 error. You can take each one in turn. (06:30) Solution step 1: What if we had two functions, one for each type of error. Each of these functions would take the entire sequence and tell us if there was an error at the beginning. Previously, our function both recognized errors and handled the sequence generation. If we pull those apart, we can add parsing for more errors easily. Each error parsing function would return nil if no error was found at the head of the sequence. (08:46) Solution step 2: Create a function that uses the two detectors to find out what error is at the head of the sequence. It takes the sequence, and wraps consecutive calls in an or block. The or block will try each one in turn until one matches and then that is the result. Each error's parsing is in its own function, and the combining function serves as an inventory. (11:35) Solution step 3: Create a lazy sequence that wraps calls to the combined detector function. Last week's code has parsing and lazy in one function. Now that we've pulled the parsing out, we can use the remaining structure to create our lazy sequence. The combined detector function is parse-next , and the function that manages the lazy sequence is parse-all . "Now we've fulfilled our obligation to have bike-shedding on naming. Next up, cache consistency. And finally, off-by-one errors." The top of parse-all has a call to lazy-seq . It will use the result of calling parse-next on the sequence. If it gets something, it will use cons to add that value to the beginning of a recursive call to itself. If it gets nil , it will recursively call itself with the rest of the sequence, thus advancing the parsing forward one step. It's not a ton of boilerplate, but it is nice to put all the mechanics of the sequence creation into a function by itself. Now we have a heterogeneous sequence of errors, and we can transform it into any report that is useful. Each parsing function doesn't need to worry about advancing down the sequence, that is handled by the higher parse-all function. Since we have a new lazy sequence, we can take it and make recognizers that take it and generate an even higher level of sequence. We ruminate more on higher level data in Episode 020. Related episodes: 020: Data Dessert 028: Fail Donut 029: Problem Unknown: Log Lines 030: Lazy Does It 031: Eager Abstraction 032: Call Me Lazy Clojure in this episode: seq , cons , rest lazy-seq or , cond Code sample from this episode: ( ns devops.week-05 ( :require [ devops.week-01 :refer [ parse-line ]] [ devops.week-02 :refer [ process-log ]] [ devops.week-03 :refer [ sprinkle-errors-by-type ]] )) ( defn parse-sprinkle [ lines ] ( let [[ first-line second-line ] lines [ _whole donut-id ] ( some->> first-line :log/message ( re-matches # "failed to add sprinkle to donut (\d+)" )) [ _whole error ] ( some->> second-line :log/message ( re-matches # "sprinkle fail reason: (.*)" ))] ( when ( and donut-id error ) ( merge first-line { :kind :sprinkle :sprinkle/donut-id donut-id :sprinkle/error error })))) ( defn parse-357-error [ lines ] ( let [[ first-line ] lines [ _whole user ] ( some->> first-line :log/message ( re-matches # "transaction failed while updating user ([^:]+): code 357" ))] ( when user ( merge first-line { :kind :code-357 :code-357/user user })))) ( defn parse-next [ lines ] ( or ( parse-357-error lines ) ( parse-sprinkle lines ))) ( defn parse-all [ lines ] ( lazy-seq ( when ( seq lines ) ( if-some [ found ( parse-next lines )] ( cons found ( parse-all ( rest lines ))) ( parse-all ( rest lines )))))) ( defn kind? ([ kind ] #( kind? % kind )) ([ line kind ] ( = kind ( :kind line )))) ( comment ( process-log "sample.log" #( ->> % ( map parse-line ) parse-all ( map :kind ) doall )) ( process-log "sample.log" #( ->> % ( map parse-line ) parse-all ( filter ( kind? :sprinkle )) sprinkle-errors-by-type )) ) Log file sample: 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | transaction failed while updating user joe: code 357 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | failed to add sprinkle to donut 23948 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | sprinkle fail reason: should never happen 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | failed to add sprinkle to donut 94238 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | sprinkle fail reason: timeout exceeded threshold 2019-05-14 16:48:56 | process-Poster | INFO | com.donutgram.poster | transaction failed while updating user sally: code 357 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | failed to add sprinkle to donut 24839 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | sprinkle fail reason: too many requests 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | failed to add sprinkle to donut 19238 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | sprinkle fail reason: should never happen 2019-05-14 16:48:57 | process-Poster | INFO | com.donutgram.poster | transaction failed while updating user joe: code 357 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | failed to add sprinkle to donut 50493 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | sprinkle fail reason: unknown state…

F

Functional Design in Clojure

1
Ep 032: Call Me Lazy 27:17

۶ years پیش27:17

27:17

Christoph finds map doesn't let him be lazy enough. Last week, we were dealing with multi-line sprinkle errors. We were able to get more context using partition . (01:33) Problem: the component lines had to be adjacent. Solution last week was to create larger partitions to hopefully get the rest of the error. This became a magic number problem, guessing how far we had to look ahead. "If there's anything I've learned in my career, telling the future is one of the hardest things to do.' What number should be big enough? 100? 1000? (04:00) The other problem is that the function is handed a pre-selected set of lines. The decision about how many lines is appropriate is made outside the function. Wouldn't it be nice if the function had control over how far to look ahead. "The function can't function." "Functions are all we've got in functional programming. Well, that and lists." It would be great if the function itself could take a sequence and look as far as it needs to. How about handing the function the entire lazy sequence? (05:52) Problem: Handing in the entire sequence means we can't use map to convert lines into sprinkle errors anymore. We can write a function that gives us just one sprinkle error from the sequence, but we want to convert the sequence into a sequence of all the sprinkle errors. We're going from something that operates on a subset of the sequence to something that operates on the entire sequence, which is too much control. We need a way for it to look ahead but still It's no longer just working on a chunk of the sequence, but on the unbounded sequence itself. We need to elevate it to the same power as other sequence operators, like map and filter . We don't, however, want the function to eagerly find all sprinkle errors in the sequence. It needs to be lazy. (09:26) Solution 1: How can we just get one sprinkle error out? If the first line isn't the error start, recur with the tail until found. Do a take-while to find the second half of the error. When both found, return the value. We need to terminate the search if we hit the end of the sequence, so we only continue if (seq lines) is not nil. "There's no sense in looking in an empty bucket." But we don't want just one, we want the entire sequence. It would be really nice to return the value when we find it and then wait to find the next one until it is requested. Conceptually, we could tell the calling function an index of where to start looking for the next error. (13:44) Solution: In Clojure, we keep our place using the lazy-seq function. lazy-seq is a sequence, but it hasn't been realized yet. It's like being able to hand back a value and a function to call for the next value. When you find a value, you can cons it onto the head of an invocation of lazy-seq to make a new sequence. Step 1. Wrap your entire function body in lazy-seq . This is similar to using delay , because it wraps the code in something that will only be evaluated when it is first accessed. Step 2. Ensure that the body obeys the contract. It must return either: nil , which indicates that the sequence is complete. a sequence, usually constructed by calling cons on a value and a call to lazy-seq . Top of the body is a call to (when (seq lines) ... , to ensure that the sequence terminates when there is no data left. Since the top of our function is lazy-seq , we can cons the found value onto a recursive call to the function. In the recursive call, we must pass the next section of the sequence, so that when evaluated it will pick up at the right place. If we don't find the start of the error, we recurse with the rest of the sequence to try parsing from there. This function will go through the sequence eagerly until it finds something. Instead of operating on single elements in the sequence, we can take a sequence and produce a sequence, powered by lazy-seq . With this capability, you can build a higher level sequence that consumes this sequence and produces a new summary, all done lazily. Related episodes: 028: Fail Donut 029: Problem Unknown: Log Lines 030: Lazy Does It 031: Eager Abstraction Clojure in this episode: partition seq , cons , rest lazy-seq , delay map , filter , take-while recur Code sample from this episode: ( ns devops.week-04 ( :require [ devops.week-01 :refer [ parse-line ]] [ devops.week-02 :refer [ process-log ]] [ devops.week-03 :refer [ sprinkle-errors-by-type ]] )) ( defn sprinkle-error-seq [ lines ] ( lazy-seq ( when ( seq lines ) ( let [[ first-line second-line & tail ] lines [ _whole donut-id ] ( some->> first-line :log/message ( re-matches # "failed to add sprinkle to donut (\d+)" )) [ _whole error ] ( some->> second-line :log/message ( re-matches # "sprinkle fail reason: (.*)" ))] ( if ( and donut-id error ) ( cons ( merge first-line { :kind :sprinkle :sprinkle/donut-id donut-id :sprinkle/error error }) ( sprinkle-error-seq tail )) ( sprinkle-error-seq ( next lines ))))))) ( comment ( process-log "sample.log" #( ->> % ( map parse-line ) sprinkle-error-seq doall )) ( process-log "sample.log" #( ->> % ( map parse-line ) sprinkle-error-seq sprinkle-errors-by-type )) )…

F

Functional Design in Clojure

1
Ep 031: Eager Abstraction 22:49

۶ years پیش22:49

22:49

Nate finds that trouble comes in pairs. Last week, we were able to make our parsing lazy. Each step was lazy, and we side-stepped the landmine of waiting to perform computation till after the file is closed. (02:00) There's a new error: Sprinkles are failing! Sprinkles are the "likes" of DonutGram, and are the metric by which all donuts are judged. Log lines are showing up, like failed to add sprinkle to donut 23948 . Let's write another regexp and handler. This is easy! But wait, we notice that there is another log line right after: sprinkle fail reason: db timeout exceeded . We want to capture that reason as part of the error, but it's on a different line. Our current code (from Episode 029), has a nifty abstraction with a paired regexp and handler function. If the regexp matches, then the matches are passed to the handler. The underlying assumption is that each line stands alone. Now we have two lines, and that breaks our pre-abstraction. Our function needs more context, give our function two lines instead of just one. Our parse-details function goes out the window, what can we do to get more context for each parsing? (07:05) Let's go to the Clojure library and check out the partition function. Used to break up a list into chunks. Eg. Chunking arg lists into key/value pairs. The step argument varies how far you reach into the collection for the next chunk. For instance, (partition 2 1 (range 1 7)) yields ((1 2) (2 3) (3 4) (4 5) (5 6)) . Each element is paired with it's following element. Doing this with our log lines means we have a second line in the group that we hand to our parsing function. The parsing function can look in the first line for the error, and then if it finds it, look in the second line for the reason. If both are found, we return a sprinkle error record. If either regex fails, nil is returned, so our list becomes a sequence of sprinkle errors or nil s. Then we can filter out the nil s and summarize the sprinkle errors. "Some of these errors hurt more than others." The sequence is not fully realized, because partition is lazy. It constructs chunks on demand. "This is the beauty of Clojure core, all this stuff is lazy. Except, of course, for the eager parts." (15:45) Problem: when looking through log lines, there are often other log lines in between. What happens if our sprinkle fail reason is not on the line immediately after the sprinkle error? We might need to look ahead a little further ahead than the next line. One solution is to increase the partition size when chunking the lines. How about (partition 10 1 lines) or (partition 100 1 lines) ? The first line is the indication that there's an error, and then look through the rest of the lines for the reason. We could log out "should never happen" if we don't find the error in the next 10 lines. It becomes a magic number problem, how far do we need to look ahead? At least these large partitions use structural sharing to re-use the record memory. We only pay extra for the containing collections. "Is there a partition-infinity ?" "It feels like something we can be lazy about, and push off to our next episode." "It's like when a show jumps the shark and you know it will come to an end, but you're just not sure which season that's going to be in." What if parsing the first line determines how many further lines we need to consume? "Fun is a loaded term." "It always puts you in an interesting situation when your pre-optimization doesn't quite hold up a month down the road." "I had the perfect abstraction for the wrong shape." "The world changed around me." "Clearly it's not the fault of my code." Related episodes: 028: Fail Donut 029: Problem Unknown: Log Lines 030: Lazy Does It Clojure in this episode: partition ->> map , filter group-by , frequencies Code sample from this episode: ( ns devops.week-03 ( :require [ devops.week-01 :refer [ parse-line ]] [ devops.week-02 :refer [ process-log ]] )) ( defn parse-sprinkle-error [ line-pairs ] ( let [[ first-line second-line ] line-pairs [ _whole donut-id ] ( some->> first-line :log/message ( re-matches # "failed to add sprinkle to donut (\d+)" )) [ _whole error ] ( some->> second-line :log/message ( re-matches # "sprinkle fail reason: (.*)" ))] ( when ( and donut-id error ) ( merge first-line { :kind :sprinkle :sprinkle/donut-id donut-id :sprinkle/error error })))) ( defn sprinkle-errors [ lines ] ( ->> lines ( partition 2 1 ) ( map parse-sprinkle-error ) ( remove nil? ))) ( defn sprinkle-errors-by-type [ errors ] ( ->> errors ( map :sprinkle/error ) ( frequencies ))) ( comment ( process-log "sample.log" #( ->> % ( map parse-line ) sprinkle-errors sprinkle-errors-by-type )) ) Log file sample: 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | failed to add sprinkle to donut 23948 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | sprinkle fail reason: should never happen 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | failed to add sprinkle to donut 94238 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | sprinkle fail reason: timeout exceeded threshold 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | failed to add sprinkle to donut 24839 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | sprinkle fail reason: too many requests 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | failed to add sprinkle to donut 19238 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | sprinkle fail reason: should never happen 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | failed to add sprinkle to donut 50493 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | sprinkle fail reason: unknown state…

F

Functional Design in Clojure

1
Ep 030: Lazy Does It 30:09

۶ years پیش30:09

30:09

Christoph's eagerness to analyze the big production logs shows him the value of being lazy instead. Last time: going through the log file looking for the mysterious 'code 357'. "The error message that just made sense to the person who wrote it. At the time written. For a few days" Back and forth with the dev team, but our devops sense was tingling. Took a sample, fired up a REPL, Ended up with a list of tuples: First element: regexp to match Second element: handler to transform matches into data (02:00) It's running slower and slower, the bigger the log file we analyze. "This is a small 4-person tech company." "Where we can use new technologies in the same decade that they were created?" "Yes!" Problem: No one turned on log rotation! The log file is 7G and our application crashes. "I think we should get lazy." "Work harder by getting lazier." "Haskell was lazy before it was cool!" Each line contains all the information we need, so we can process them one at a time. (4:30) Eager and lazy is like the difference between push and pull. Giving someone a big bag of work to do, or having them grab more work as they finish. The thing doing the work needs a little more to work on, so it pulls it in. Clojure helps with this. It gives us an abstraction so we don't have to see the I/O happening when we do our processing. When your map of a lazy sequence needs more data, it gets it on demand. Clojure core is built to support this lazy style of processing. File I/O in Java is lazy in the same way. It reads data into a buffer and when that buffer is used up, more is read. Lazy processing is like a bucket brigade. At the head, you pour out your bucket and the person next to you notices the empty bucket and fills it up. Then this is repeated down the line as each bucket demands to be filled. (07:55) Let's make our code lazy. Current lines function slurps in the file and splits on newline. Idea: Convert it to open the file and return a lazy sequence using line-seq . The return value can be threaded through the rest of our pipeline. Each step of our pipeline is lazy, and the start is lazy, so the whole process should be lazy. "It's lazy all the way." Problem: We run it, and BOOM, we get an I/O error. What happened? We were using the with-open macro, which closes the file handle after the body is complete. Since the I/O is delayed till we consume the sequence, when we start the file is already closed. "The ability to pull more I/O out of the file has been terminated." "Nothing at all, which is a lot less useful than something." (12:29) Rather than having a lines function, why don't we just bring that code into the summary function? Entire process is wrapped in a with-open so that all steps including summary complete before the file is closed. Takes a filename and returns an incident count by user. It does all that we want, but it's too chunky. We're doing too much in the function. We usually want to move I/O to the edges and this commingles it with our logic. "We just invited I/O to come move into the middle of our living room." I/O and summary are in the same function, so to make a new summary, we have to duplicate everything. We could split out the guts, extract the general and detailed parsing into a separate function. For reuse. This means you are only duplicating the with-open and line-seq for each new summary. (16:29) How can we stop duplicating the with-open ? To separate that idiom into just one place. If you can't return the line-seq , is there a way we can hand in the logic we need all at once? Idea: Make a function that does the with-open and takes a function. "Let's make it higher order." We hand in the "work to do" as a function. "What should we call it? How about process . That's nice and generic." We turn the problem of abstraction into the problem of writing a function that takes a line sequence and produces a result. Any functions that take and produce a sequence, including those that we wrote, can be used to write this function. Clojure gives us several ways of composing functions together, we'll use ->> (the thread-last macro) in this case. As we improve the vocabulary for working with lines, our ability to express the function gains power and conciseness. Design tension: if there is something that needs to be done for every summary, it can be pushed into the process function. The downside to that is that we sign up for that for every summary, and that might not be appropriate for the ones we haven't written yet. We opt for making it easier to express and compose the function passed in. We can still make a function that takes a filename and returns a summary, but the way we construct that function is through composition of transforms. We can pre-bake a few transforms into shorter names if we want to use them repeatedly. (23:20) We will still run into the I/O error problem if we're not careful. The function that we pass to process needs to have an eager evaluation at the end. If all we do is transform with lazy functions, the I/O won't start before the list is returned. group-by or frequencies will suffice, but if you don't have one of those, reach for doall . "You gotta give it a good swift kick with doall ." Style point: doall at the beginning or at the end of the thread? We like it at the end. (26:07) We have everything we need. Lazy so we don't pull in the entire file. I/O sits in one function. We have control over when we're eager. Message Queue discussion: (26:38) Long-time listener Dave sent us a code sample! An alternative implementation for parse-details that doesn't use macros. Top level is an or macro. Inside the or , each regex is matched in a when-let , the body of which uses the matches to construct the detailed data. If the regex fails to match, nil is returned and the or will move on to the next block. We tend to think of or as only for booleans, but it works well for controlling program flow as well. The code is very clean and concise. And it only uses Clojure core. "Without dipping into macro-land... Not that there's anything wrong with that." Related episodes: 028: Fail Donut 029: Problem Unknown: Log Lines Clojure in this episode: slurp , with-open , line-seq ->> or , when-let map , filter group-by , frequencies doall clojure.string/split-lines Code sample from this episode: ( ns devops.week-02 ( :require [ clojure.java.io :as io ] [ devops.week-01 :refer [ parse-line parse-details ]] )) ; Parsing and summarizing ( defn parse-log [ raw-lines ] ( ->> raw-lines ( map parse-line ) ( filter some? ) ( map parse-details ))) ( defn code-357-by-user [ lines ] ( ->> lines ( filter #( = :code-357 ( :kind % ))) ( map :code-357/user ) ( frequencies ))) ; Failed Attempt: returning from with-open ( defn lines [ filename ] ( with-open [ in ( io/reader filename )] ( line-seq in ))) ( defn count-by-user [ filename ] ( ->> ( lines filename ) ( parse-log ) ( code-357-by-user ))) ; Throws IOException "Stream closed" # _ ( count-by-user "sample.log" ) ; Works, but I/O is coupled with the logic. ( defn count-by-user [ filename ] ( with-open [ in ( io/reader filename )] ( ->> ( line-seq in ) ( parse-log ) ( doall ) ( code-357-by-user )))) # _ ( count-by-user "sample.log" ) ; Separates out I/O. Allows us to compose the processing. ( defn process-log [ filename f ] ( with-open [ in ( io/reader filename )] ( ->> ( line-seq in ) ( f )))) ; Look at the first 10 lines that parsed # _ ( process-log "sample.log" #( ->> % parse-log ( take 10 ) doall )) ; Count up all the "code 357" errors by user ( defn count-by-user [ filename ] ( process-log filename #( ->> % parse-log code-357-by-user ))) # _ ( count-by-user "sample.log" ) Log file sample: 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | transaction failed while updating user joe: code 357 2019-05-14 16:48:56 | process-Poster | INFO | com.donutgram.poster | transaction failed while updating user sally: code 357 2019-05-14 16:48:57 | process-Poster | INFO | com.donutgram.poster | transaction failed while updating user joe: code 357…

F

Functional Design in Clojure

1
Ep 029: Problem Unknown: Log Lines 35:06

۶ years پیش35:06

35:06

Nate is dropped in the middle of a huge log file and hunts for the source of the errors. We dig into the world of DonutGram. "We are the devops." Problem: we start seeing errors in the log. "The kind of error that makes total sense to whoever wrote the log line." (04:10) We want to use Clojure to characterize the errors. Why not grep? Well...we grep out the lines, count them, and we get 10,351 times. Is that a lot? Hard to say. We need more info. (05:50) Developers think it's safe to ignore, but it's bugging us. We want to come up with an incident count by user. "Important thing we do a lot in the devops area: we see data and then we have to figure out, what does it mean?" "What story is the data telling us?" The "forensic data" versus "operating data" (Ep 022, 027) "With forensic data, you're trying to save the state of the imprint of the known universe over time." (07:40) The log file is the most common forensic data for applications. To characterize, we need to manipulate the data at a higher level than grep. First goal: make the data useful in Clojure Get a nice sample from the production log, put it in a file, load up the REPL, and start writing code to parse it. Need to parse out the data from the log file and turn it into something more structured. (12:05) Let's make a function, parse-line that separates out all the common elements for each line: timestamp log level thread name package name freeform "message" "A message has arrived from the developers, lovingly bundled in a long log line." (13:10) We can parse every line generically, but we need to parse the message and make sense of that. "We want to lift up the unstructured data into structured data that we can map, filter, and reduce on." We will have different kinds of log lines, so we need to detect them and parse appropriately. We want to amplify the map to include the details for the specific kind of log line. We'll use a kind field to identify which kind of log line it is. There are two steps: recognize which kind of log line the "message" is for parse the data out of that message "Maybe we'll have a podcast that's constantly about constantly . It's not just juxt ." (18:45) How do we do this concisely? Let's use cond : flatten out all the cases code to detect kind (the condition) code to parse the details (the expression) Can use includes? in the condition to detect the kind and re-matches to parse. (20:00) Why not use the regex to detect the kind too? Can avoid writing the regex twice by using a def . It's less code, but now we're running the regex twice: once in the condition and once to parse. We can't capture the result of the test in cond . No "cond-let" macro. We could write our own cond-let macro, but should we? "When you feel like you should write a macro, you should step back and assess your current state of being. Am I tired? Did I have a bad day? Do I really need this?" (24:05) New goals for our problem: one regex literal only run the regex once keep the code that uses the matches close to the regex Similar problem to "routes" for web endpoints: want the route definition next to the code that executes using the data for that route. Feels like an index or table of options. (25:20) Let's make a "table". A vector of regex and handler-code pairs. We need a coffee mug: "Clojure. It's just a list." The code can be a function literal that takes the matches and returns a map of parsed data. Write a function parse-details which goes through each regex until one matches and then invokes the handler for that one. (See below.) (30:15) Once we have higher-level data, it's straight forward to filter and group-by to get our user count. Once again, the goal is to take unstructured data and turn it into structured data. "You have just up-leveled unstructured information into a sequence of structured information." Can slurp , split , map , filter , and then aggregate. (32:10) What happens when we try to open a 10 GB log file? Sounds like a problem for next week. "Fixing production is always a problem for right now." "If a server falls over and no one outside of the devops team knows about it, did the server really fall over?" "Who watches the watchers?" Related episodes: 020: Data Dessert 022: Evidence of Attempted Posting 027: Collected Context Clojure in this episode: #"" re-matches case cond if-let ->> slurp filter group-by def constantly juxt clojure.string/ includes? split Related links: GraalVM Code sample from this episode: ( ns devops.week-01 ( :require [ clojure.java.io :as io ] [ clojure.string :as string ] )) ;; General parsing ( def general-re # "(\d\d\d\d-\d\d-\d\d)\s+(\d\d:\d\d:\d\d)\s+\|\s+(\S+)\s+\|\s+(\S+)\s+\|\s+(\S+)\s+\|\s(.*)" ) ( defn parse-line [ line ] ( when-let [[ whole dt tm thread-name level ns message ] ( re-matches general-re line )] { :raw/line whole :log/date dt :log/time tm :log/thread thread-name :log/level level :log/namespace ns :log/message message })) ( defn general-parse [ lines ] ( ->> lines ( map parse-line ) ( filter some? ))) ;; Detailed parsing ( def detail-specs [[# "transaction failed while updating user ([^:]+): code 357" ( fn [[ _whole user ]] { :kind :code-357 :code-357/user user })] ]) ( defn try-detail-spec [ message [ re fn ]] ( when-some [ matches ( re-matches re message )] ( fn matches ))) ( defn parse-details [ entry ] ( let [{ :keys [ log/message ]} entry ] ( if-some [ extra ( ->> detail-specs ( map ( partial try-detail-spec message )) ( filter some? ) ( first ))] ( merge entry extra ) entry ))) ;; Log analysis ( defn lines [ filename ] ( ->> ( slurp filename ) ( string/split-lines ))) ( defn summarize [ filename calc ] ( ->> ( lines filename ) ( general-parse ) ( map parse-details ) ( calc ))) ;; data summarizing ( defn code-357-by-user [ entries ] ( ->> entries ( filter #( = :code-357 ( :kind % ))) ( map :code-357/user ) ( frequencies ))) ( comment ( summarize "sample.log" code-357-by-user ) ) Log file sample: 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | transaction failed while updating user joe: code 357 2019-05-14 16:48:56 | process-Poster | INFO | com.donutgram.poster | transaction failed while updating user sally: code 357 2019-05-14 16:48:57 | process-Poster | INFO | com.donutgram.poster | transaction failed while updating user joe: code 357…

F

Functional Design in Clojure

1
Episode 013: Connect the REPL 32:10

۶ years پیش32:10

32:10

Nate continues to explore the REPL by gluing it to his editor to see what happens. We revisit Tic-Tac-Toe and change it using our new REPL superpowers. Let's add request logging to the play endpoint: /play?row=0&col=1 . Need to add log/info calls in handler function for play endpoint: on-play . Naive Christoph would add logging to the function, recompile, and run the jar. Less naive Christoph would reload the namespace but still restart the web server. Crazy idea: paste the function definition into the REPL! Copy the whole defn for on-play Go to the REPL window Switch to the app.main namespace (where we put on-play ) Paste the function definition into the REPL No restarting required! Each time we evalute the defn in the REPL, the new function replaces the old version immediately! We can iterate on a single function quickly. Clojure is Lisp, and Lisp is dynamic and isomorphic Dynamic: can change definition on the fly, so the new function exists immediately after executing the definition Isomorphic: everything is a "form". No special "declaration syntax" vs "expression syntax". A declaration is a form just like any other expression. "Clojure is dynamic to the end. Every statement is modifying the running application, and because every statement has access to the entire application, all the way down to the definitions, you can redefine on the fly." Declarations are just ordinary macros ( def , defn , defmacro , etc), not special syntax only the compiler understands. A declaration macro updates the enclosing namespace dynamically on the fly! "You just replaced the wheels on the car while you were driving." "I can replace the steering wheel and my hands don't even realize they're using a new steering wheel." "Running the form is what brings that thing dynamically into existence or replaces the one that was there." "It's the way you can speed your development up because you can replace these functions on the fly." Problem: you have to copy and paste between the editor and the REPL. Idea: use terminal automation to help with the copy + paste! "Yet another example of solving a problem you probably shouldn't even have." Pre-Lisp thinking: "The right way to do software is to compile it from grains of sand each and every time, and then run it from the beginning." Better Idea: Connect the REPL directly to your editor. (See "More reading" for how.) How it works: Move your cursor to the form to evaluate Do the "right" key combo (depends on your editor) Editor sends the form to the REPL REPL evaluates the form and sends the result back Editor shows the result Use comment blocks as a nifty way to provide executable examples within the code. Your editor is a first-class REPL. Related episodes: 004: Atomic Curls 012: Embrace the REPL More reading: Enhancing your REPL workflow Guidelines for REPL-Aided Development Clojure in this episode: ns , in-ns , use *ns* , ns-map def , defn comment let println clojure.tools.logging/debug Related projects: clojure.tools.logging for application logging vim-fireplace for VIM connected REPL Cider for Emacs connected REPL Ring for HTTP Compojure for HTTP request routing…

F

Functional Design in Clojure

1
Episode 012: Embrace the REPL 25:33

۶ years پیش25:33

25:33

Christoph complicated development by misunderstanding the REPL. We go back to the roots of Christoph's experience with Clojure... How do I run Clojure code? The REPL is fun for evaluation, but how do you run a "real" program? Experience in other languages: edit the file, compile, rerun the program Mentality: get through the edit, compile, run loop as fast as possible! Autobuilder is the logical end. "Where's my autobuilder in Clojure?!" The REPL model is fundamentally different. "[The REPL] is like cutting the Gordian Knot. You change the problem and that's how you solve it." "The reason why the problem I wanted solved, wasn't 'solved', is because nobody has that problem because they do things differently here." Tactic 1 : edit, save, restart the REPL "Restarting the REPL isn't super slow, but let's just say it's not instantaneous." Tactic 2 : edit, save, run (use 'the.namespace :reload) in the REPL "Now I have a REPL history of use statements!" Problems: forgetting to reload dependent namespaces loading dependencies in the wrong order old definitions built up Big Problem: function renames left the old version around, so accidental calls using the old name produced no errors and ran old behavior! Back to quitting the REPL to clean out the cruft. Ugh! Discovered clojure.tools.namespace ! Reloads everything and cleans out the cruft! Tactic 3 : edit, save, (use '[clojure.tools.namespace.repl :only (refresh)]) , (refresh) Problem: refresh would purge itself! "I don't know why it took me so long to discover the magical user namespace!" The REPL will automatically use the user namespace. "I can put code into a user namespace and it will be there for me." Christoph would switch namespaces in the REPL without even stopping to wonder what namespace he started in. "It's like walking out of a door and not even thinking about the fact you're in a building. Oh wait! What room did I just leave?" Tactic 4 : make sure refresh is in the user namespace, edit, save, (refresh) However, Christoph still didn't understand the REPL. Just thought the REPL was for: reloading the code and restarting evaluating snippets of code inspecting stuff Nate's big ah-ha moment: "Not only is my application inspectable, it's fungible. I can change it in place as it's flying along! I don't have to restart it!" Christoph's hint that there was still more came from seeing comment blocks in code and reading about vim-fireplace. "There's a new room you can explore. That room is editor integration with the REPL." Clojure in this episode: use clojure.tools.namespace.repl/refresh clojure.pprint/pprint Related projects: clojure.tools.namespace vim-fireplace Cider for Emacs…

F

Functional Design in Clojure

1
Episode 011: The Convention of Configuration 28:47

۶ years پیش28:47

28:47

Nate is worried about the hardcoded credentials in the code. It's episode 011 on 01/11. It's binary! "As a developer, you quickly learn what's under your control and what's not. Very little is under your control." Don't accidentally check in the credentials. "We need a configuration management system. Oh wait! That's a totally different problem." What about putting the configuration into an EDN file? Let's call it config.edn Why EDN? Isn't JSON the "one, true format for config information"? Pros of EDN: native to Clojure can have comments is extensible Why not environment variables? Environment variables are great for production, but in development files are better. Have to restart the REPL to change env variables. Make a component that reads the config. That will reload config when you reset with component.repl. Two main considerations: Dynamically reloading configuration during development Plumbing the configuration values through the app Make a namespace for app.config "Files are way more fungible than the environment." We want both options: env for production and a file for dev. Make two functions in app.config : from-env and from-file . Use schema to make sure both functions return the same config map. "A thing I have done before...you decide if it's clever." Define a default configuration and then merge the config maps with that. Better yet, from-env handles defaults and we merge the map from dev.edn into that. Can use environ with Leiningen profiles. Still requires restarting the REPL. Defaults should make sense for development, so you can just check out and run. For bad config, we want helpful error messages, not stack traces. What goes in configuration? Items under your control Items that vary Twitter API URL does not vary. Even if Twitter provided sandbox URLs, those URLs don't vary. The config option should be "production" and "sandbox", not the URL. The wrapper code will select the right URL. Avoid the nonsense of trying to infer "sandbox" from reading the URL. "The hallmark of good design is: the less thinking you have to do, the better." "The more semantic the config, the less thinking you have to do." It's important to think about the data first, where it comes from, and where it goes. Clojure in this episode: merge try , catch slurp clojure.edn/read-string environ.core/env component.repl/reset Related projects: Component component.repl Environ Leiningen , profile support Schema Code sample from this episode: (ns app.config (:require [clojure.edn :as edn] [environ.core :refer [env]])) (defn from-env [] {:twitter-key (or (env :twitter-key) "") :twitter-secret (or (env :twitter-secret) "") :initial-tweets (or (env :initial-tweets) 20)) (defn config [] (merge (from-env) (edn/read-string (try (slurp "dev.edn") (catch Throwable e "{}")))))…

F

Functional Design in Clojure

1
Episode 010: From Mud to Bricks 28:35

۷ years پیش28:35

28:35

Christoph can't stand the spaghetti mess in main. Time to refactor. Main function does too much. We want cohesive pieces that each have one job. Two part problem: Too much in the main loop. Main starts and then never stops. Have to exit the repl to restart. We need 1) separate parts that can be 2) started and stopped. Main should just focus on the code needed for command line invocation. Let's get the parts component-ized! "It's one thing that I've become more sure of in my career is that no application is actually done." "Useful apps only get more complex." Internal dependencies are different than external dependencies ("libraries"). Many internal dependencies create high coupling throughout the code. "Once everything starts touching everything you have to understand everything to understand anything." Like functions use parameters to limit scope, a component is the next level up and uses resource dependences to limit coupling. Each component implements a clear behavior (interface) and can be a resource to other components. Can understand component's behavior via its interface (and docs) without reading all the code--just like understanding a function through it's signature and docs. We use the "Component" library to make components in Clojure--has REPL integration too. Components to make: web server polling and fetching loop the core.async channel used between them The Lifecycle interface allows a component to be started and stopped. start must return a reference to the "started" state stop must return a reference to the "stopped" state Gotcha: don't return a nil ! To use Lifecycle , you'll need to make your component a "record". Two main goals of Component: allow stateful components define dependencies between components Surprise! Any reference can be a "component" as a non-lifecycle dependency. Write a function new-system which returns the component "system map" Mind your names. Make the system map key match a component's dependent field. "Component is a convenient way of being able to specify all those dependencies in a concise map, so you know this is the intersection of all of my application together." A component should be able to be understood alone. "Component is like giving you application parts as function parameters. Just like when making a function, you don't worry about how something gets passed in as a parameter." You still need to understand each of the parts, but you don't have to worry about where the part came from. At the top level, you can see all the parts together and how they are connected. Immutability gives you bulkheads between each of your components so you can safely reason about them separately. Use component.repl to start and stop the whole system without restarting the REPL! Need some tooling to keep the main thread from exit. Can use promise , deref and a shutdown handler (see below). "We can keep each ball of mud it its own little basket so all the mud doesn't ooze together." Clojure in this episode: defrecord promise , deliver deref , @ component/ Lifecycle system-map using start , stop component.repl/ go stop reset Related projects: Component component.repl Code sample from this episode: (ns app.main (:require [clojure.core.async :as async] [com.stuartsierra.component :as component] [app.component [fetcher :as fetcher] [web :as web]]) (:gen-class)) (defn new-system [] (component/system-map :new-search-chan (async/chan) :web (component/using (web/new-component) [:new-search-chan]) :fetcher (component/using (fetcher/new-component) [:new-search-chan]) (defn -main [& args] (let [system (component/start (new-system)) lock (promise) stop (fn [] (component/stop system) (deliver lock :release))] (.addShutdownHook (Runtime/getRuntime) (Thread. stop)) @lock (System/exit 0)))…

F

Functional Design in Clojure

1
Episode 009: Channeling Re-Search 26:47

۷ years پیش26:47

26:47

Nate can't decide what to watch on Twitter, and the app restarts are killing him. The Twitterthon continues. "It's an infinite stream of Twitter." Nate wants to watch #clojurescript too. Just change the search string to "#clojure OR #clojurescript" ? Should avoid hardcoding the value so we don't have to recompile every time. Command line argument still requires a restart. Let's use a curl UI (like Ep 004)! Wait, what?! Why curl ? Can send something into the running process Can separate out the client Builds on top of HTTP, so there are lots of tools for testing (like curl itself) Use a URL query string: http://localhost:8000/search?The+search+string "It's a true query string in the truest sense of the term 'query.'" "It is actually using the thing for the thing it was meant to be." How do we get the query string from the webserver thread to the polling loop thread? "This is a perfect case for a global." Oh wait, that's mutation. How should we structure the app? The main function must: start the thread for the web server start the thread for the polling loop Specific problem: the /search route handler function needs a way to send the new query string to the polling loop. With mutation: share an atom and give the handler and the polling loop a reference to it. No mutation? Use a core.async channel. A channel allows a thread to pass data to another thread. With no channel buffer, it will synchronize two threads ("rendezvous"). (We lie. There is no peek function.) Problem: polling thread is stuck waiting on the channel, so it stops polling. Solution: Use alt!! to simultaneously listen to the "new search" channel and a timeout channel. What is a timeout channel? A channel that closes after n msecs have passed. New problem: the cache (for de-duplicating) has stale content after changing the query. Solution: same logic that adopts the new search term will reset the cache. (See Ep 007.) Polling loop structure: fetch process results block while listening to the channels if new search string, recur with new cache and new search string otherwise, recur with same cache and search string Only want the fetch in one part of the loop. Don't even need curl . Just type in the URL for the backend on your phone. core.async lets threads coordinate without having to know about each other! Clojure in this episode: atom loop , recur core.async/ chan , <, >! , >>! , put! alt! , alts! timeout Related projects: aleph and http-kit core.async compojure…

F

Functional Design in Clojure

1
Episode 008: Twitter, Plated 23:07

۷ years پیش23:07

23:07

Christoph tries to figure out why Twitter stopped talking about Clojure. "Are you twitterpated?" Building on where we left off last episode. Runs and just stops working. "I was pretty sure it stopped working because people on Twitter just stopped talking about Clojure. After about a day of that, I realized people were talking about Clojure, and I just wasn't seeing it." The auth token expired! What should we do? Why should the main loop have to deal with getting a new auth token? "The Twitter wrapper should be concerned with all of the warts and complexities of dealing with Twitter." "What problems should bubble up, and which ones shouldn't?" The wrapper should handle the retry. It's like a kitchen in a restaurant. What are the steps of fulfilling an order? The customer doesn't care. "There's a side-effect: the freezer mutates." The wrapper gets to worry about all the steps: turning the order into the specific request for the kitchen do the I/O to fetch and fulfill the request the "input transform" takes the mass of data and picks out the relevant parts the "internal" version is returned "Like all good metaphors, they stretch to the point where they break, like a rubber band." Maybe avoid expired tokens by authenticating every time? Too much overhead. If the handle is mutable, then retry can just update the handle with the new token. A mutable handle does allow the wrapper to control the concern. The "handle" is the state of the wrapper. The term "handle" comes from I/O libraries. Instead of mutation, have the search function return [updated-handle, result] . search can catch an auth exception, retry, and return a new auth handle. Instead of search retrying, the fetcher can do it! Then it works for all kinds of requests. Better yet, leave fetch simple, and have a fetch-with-retry function that uses fetch . Can have even more policy functions like, fetch-with-retry-forever . "Keep calm, and assoc on." "I'm never going to miss another Clojure tweet. I'm going to read them all!" Clojure in this episode: loop , recur try , catch atom assoc…

F

Functional Design in Clojure

1
Episode 007: Input Overflow 26:34

۷ years پیش26:34

26:34

Nate just wants to print the tweets, but the input wants to take over. Got tweet fetching working last time. Now we want to print out the tweets. API returns a lot information "We should probably not use pprint as our output mechanism." Using the uniqueness filtering from last time The goal is NOT to write a generic Twitter wrapper. Goal is to write a wrapper that just gives us what our application needs. "I don't want to spread the complexity of Twitter all over my app. I want to hide it inside of the wrapper." Clojure data structures are generic enough already! We pick out just what we need and put it in our own data structure. We use our "internal" structure throughout the application code. Our internal structure protects our application code from changes in Twitter's format. Keep the structure minimal and grow it when the application code needs more fields. Where should we put the internal model? Make it a part of the wrapper. A wrapper should expose a data model A "sequencing" function just threads steps defined by other functions. Pure steps are easy to test, so make the I/O steps as minimal as possible. Technique is called "pushing I/O to the edges". Let any exceptions fly back to the main loop. It can sleep a little and retry. Separate out formatting from println , so you can unit test output. Put the cache behind its own data model The cache data model provides logical operations like (defn filter-new [cache tweets] ...) The filter-new function would return [cache new-tweets] . Cache could have changed. How do you know what's in the data model exposed by the wrapper? "What is in this thing that we're passing around? You don't just want to read the code and figure it out?" "You rapidly get to the point where you don't know what's in your data structure." Use schema. Put the schemas the wrapper exposes in its own namespace. Parts of a data model the data structure: what that model needs to store or represent a schema (or spec) to define and validate the data structure "constructor" functions that take args and return a data structure "update" functions that take the data structure and return a modified one "output" functions that take the data structure and return some other representation Parts of a wrapper Application-facing functions. Sole job is to sequence internal wrapper functions. Data model for a Twitter API "request" Functions that execute a request and return raw data Data model exposed by the wrapper Clojure in this episode: println get-in -> loop , recur , reduce lists as "tuples" schema/defschema , schema/Int , schema/String , ... Related projects: core.cache schema…

F

Functional Design in Clojure

1
Episode 006: All Wrapped Up in Twitter 25:03

۷ years پیش25:03

25:03

Christoph tries to get a handle on his #clojure tweet-stream habit. NOT tic-tac-toe Follow #clojure tweet stream and see it print out in the terminal "We like reinventing things." "The terminal is the best UI out there." Does Twitter have an API? Websocket? Nope! Requires a big "E" plan: "enterprise". PubSub? Nope! Not from Twitter. Alas, we must poll the /search/tweets.json endpoint Problem: we're going to keep getting results we've already seen Avoid duplicates? Let's use core cache. Once again, we use loop and recur for our main loop Time for an API wrapper, but what does the wrapper do? HTTP POST form-encoded parameters Ack! 401 Access Denied "An important step in any API exploration is your first 401 response." "OAuth?" "Oh....auth..." Meet OAuth, the API bouncer. Make an auth function to call the OAuth endpoint and get an auth token Have auth return a "handle" with the auth token. Other wrapper functions will need handle . Need to keep handle around. Put that in the app state too. Let the exceptions fly! "Exceptions are an exceptionally accepted way of handling exceptional circumstances." "I caught what you meant." Make a fetch function that does the I/O work. Create a search function that takes handle and query Look for pure logic and move it into its own function, then it's easy to test. Transform args to search into a "request description" and have fetch operate on that. "Twitch, I mean Twitter. You know, that Internet thing that starts with t-w-i-t." Different layers of the wrapper: Top-level functions used by the application. Sole job: sequence internal functions Pure transforms from args to "request description" A fetch function that follows the "orders" of the "request description" "The point of testing this code is not to test if Aleph works or Twitter works. The logic is in the translation function, so that's why we test it." "Our hero, the Twitch wrapper, is poised and ready to fetch data on our behalf. What will happen next?" Clojure in this episode: loop , recur Thread.sleep get try , catch -> Related projects: core.cache Aleph and http-kit Manifold Netty…

F

Functional Design in Clojure

1
Episode 005: Tracking, for the Win 26:19

۷ years پیش26:19

26:19

Nate tries to figure out who actually won this never-ending game of tic-tac-toe. Tic-tac-toe is "just boring enough, just interesting enough." How do we know who won the game? Inspect the board. If you can track progress toward the win, you check for the win quickly "Tic-tac-toe at scale!" Tracer bullet: go simple, just examine the 8 options for winning "In that case, nil has won...which is nobody." Keep detection logic out of the high-level winner function--should read like a process description of steps. Make new "verbs" like row-winner and column-winner and use those. "You're just adding new verbs to raise Clojure up to the level of your problem. You can speak about your problem using those verbs." Let's make it faster! Need incremental detection to be efficient. Tracking structure with win condition totals keys like: [ player case index ] value is a counter for that case eg. { [:x :row 0] 1, [:y :row 0] 0, [:x :diag] 2, ...} The win tracker is a "nested model" of the game state Put the tracker it its own namespace app.game.tracker Use [:x 1 0] as the play Nested updates: (update game-state :tracking tracker/update [:x 1 0]) How do we handle diagonals? Not all moves will increment those totals. Make helpers for diag? and rdiag? to use in cond-> (see code below) High-level functions describe the process. Low-level functions describe the steps. "You can see the animal, not the intestines." "If you see a word that's a higher level concept, it allows you to stay at that higher level and be able to view the algorithm instead of viewing the implementation. That's the point of lifting up all these little functions." Bonus: the tracker tells us all the ways a player won. Clojure in this episode: nil punning streak: 3 episodes get-in update , update-in or short circuits, = does not -> , cond-> , some-> frequencies lists as "tuples" and "triples" Code sample from this episode: (ns app.game.tracker) (defn update [tracking [player row column]] (cond-> tracking true (record row column player) (diag? row column) (record-diag player) (rdiag? row column) (record-rdiag player)))…

F

Functional Design in Clojure

1
Episode 004: Atomic Curls 22:40

۷ years پیش22:40

22:40

Christoph tries to make tic-tac-toe work over the Internet and discovers the power of the atom. Let's get a web framework, build a UI, hook that up to an HTTP API, throw in some websockets for notifications! "We'd end up with our first 12 hour podcast episode." A "tracer bullet": get something functioning to see the important parts of the solution. Can choose when to replace important parts by something more "production worthy". "Let's just ditch all the complexity of having a UI and just use curl ." curl will print the text response. We have a terminal UI! "We're extending the command line out to the web!" "I just keep curling every day or every other day until it's my turn." "It's your morning curls!" How do we handle web requests? What is a route? Routes: /new , /show , /play?row=0&col=1 "Super simple URL API. Who needs REST?" Let's run this on port 1337 to hide it from the Internet. One shared game state stored in an atom. It's the only game in town! Use an atom instead of a database to cut complexity while problem solving. The ! ("bang") in swap! and reset! indicates you're causing a side effect. Handler's sole job: take web requests and alter the game state using the game model. Function called by swap! should be pure. Don't throw exceptions! Dilemma: How much do you put inside the function called by "swap!"? For errors, the transaction function can: Return unchanged reference. Use "triage" function to diagnose. Have an "error" attribute in the state and set an "outcome" code like :success , :invalid-coordinate , etc. Tracer bullet shows lots of ways to complex-ify this into "Tic-Tac-Toe, Enterprise Edition" Clojure in this episode: defonce atom deref swap! and reset! aleph and http-kit compojure…

F

Functional Design in Clojure

1
Episode 003: Tic-Tac-REPL 23:09

۷ years پیش23:09

23:09

Nate tries to turn the tic-tac-toe "game engine" into a real application he can play with a friend. Let's play the game! How do you keep track of the game state as it changes? Bucket brigade the reference to the future loop using recur The game loop: read input, evaluate, print out new board, loop. "It has occurred to me that we are basically writing a REPL." "We have the tic-tac-REPL" How do you get input from the user? How to you make sure it's right? "It keeps harassing the non-compliment user until they type the right thing in" Input loop: read, validate, loop on error, return on success Keep the logic pure! Separate out the parsing and validation functions. "Much better to tuck it away in a function!" Sequence the pure parts with a minimalist function that does the I/O. "I don't like having ovens present because they're hard to put in my test cases." Unit test those pure parts. (No one likes to be mocked.) I/O is a side effect! "Every time I redefine one of those things I feel like I'm reaching down into the bowels of Clojure and doing something moderately illegal." Using keywords as error codes is nifty "You don't have any different kinds of nil . You just have one. It's the nuh-uh." Can use a tuple with the first element always being a keyword and the second being data for the "details" Clojure in this episode: read-line string/split swap! and reset! loop and recur let vs loop keywords nil punning…

F

Functional Design in Clojure

1
Episode 002: Tic-Tac-Toe, State in a Row 20:00

۷ years پیش20:00

20:00

Christoph tries to make tic-tac-toe and gets stuck on immutability. The REPL calculator Let's make a game! How do you keep track of the game board? How do you update the game board if you can't change anything? OO told me the "right" way to encapsulate, now what do I do? "Nine lines of let block, and one line of actual function." "The reference bucket brigade" reductions wants to blow your mind Multiple universes of tic-tac-toe, and they're all in a row! Time travel, for free! Clojure in this episode: assoc assoc-in -> reduce reductions…

F

Functional Design in Clojure

1
Episode 001: Why, Oh Why? 13:44

۷ years پیش13:44

13:44

Nate and Christoph try to figure out how to make a podcast. Who are we? What are we doing? What will we talk about? Lots of (with-meta podcast ...) Clojure in this episode: nil

F

Functional Design in Clojure

1
Ep 028: Fail Donut 16:56

۶ years پیش16:56

16:56

Christoph has gigs of log data and he's looking to Clojure for some help. Introducing a new topic. The last few weeks we focused on Twitter and automatically posting to it. Surprised by how much there is to talk about in a focused problem. "There will always be more problems for the world to solve." (01:53) Imagine if you will, the world of DonutGram! A fictitious social network where people post their donut experiences. This series is not about making DonutGram, but living DonutGram. "Oh, the dark underbelly of application development: support." "Let's take the shiny rock and flip it over and look at all the worms and bugs." We talked about forensic data before, and much of DevOps is about looking at that data. You want to paint the most complete story. It's a development problem too. Most of the time, forensic data is written to a log file. It's the first line of investigation. Imagine all of the components of your application speaking into one pipe, and you now get to reconstruct what happened. (05:59) Everything is rosy at DonutGram, but then users start having issues. Users get the "Fail Donut" and start posting that. We will be assuming the role of the heroic DevOps team. There is quite a bit of data written to the log file, more and more with each bugfix. We're going to use Clojure to investigate this problem. (08:50) What problems might we encounter? Problem 1: Size No one configured log rotation! The log file is too large to load for analysis. Problem 2: Unstructured data Everything is a line or multiple lines. We can build up layers of abstractions to gradually build understanding. Problem 3: Non-linear data We want to tell a story about what went wrong. Many times, the pieces of that story are in different areas of the log file, and must be collated. There can be multiple competing stories. The parts of the story are like dominoes. How do you know if the third domino is important without knowing that the first two have fallen? Problem 4: Alerts "If you have a really good story to tell, how do you tell anyone about it?" Alerts should be timely, depending on the audience. Call out to the audience: do you have battle stories about processing log files? Let us know! Clojure in this episode: nil…

F

Functional Design in Clojure

1
Ep 027: Collected Context 33:20

۶ years پیش33:20

33:20

Nate and Christoph reflect on what they learned during the Twitter series. 6 months of podcast episodes! Situated programs: Part of the real world. Affect and affected by the world around it. (04:25) Concept 1: Was very helpful to focus on nailing down the algorithm before diving into the code. (06:00) Concept 2: Two kinds of data: operating data and forensic data Operating data: information you use for logic. Make it aggressively minimal. Forensic data: information you record, off to the side, to tell a story later. Record as much as feasible. Don't do logic on the forensic data! If you end up writing code to sift through the forensic data in the future, cherry pick out the data need to use and treat that as operational data for that problem. "'No' is temporary. 'Yes' is forever." (09:45) Concept 3: Use components to wrap integrations We wrapped DB and Twitter integrations. If a component needs a resource, that resource should be a component. Because resources are components, you can substitute the real component with a "faker" component. (11:10) Concept 4: Fake resources for productivity A "faker" component takes the place of a real component to enable developers to be more productive. Faking is not for testing code, it's for effective development. Can avoid Internet and latency. Can run the code right out of the repo. Allows you to explore how the whole system behaves during edge cases and behaviors. Allows you to start with useful information or quickly reset back to scenarios. "If an edge case is going to be annoying in the wild, you might as well get used to it being annoying during development." (13:40) Concept 5: Aggressively decouple decision logic from imperative work Push side effects (like I/O) to the "edges", but some situations make that less straightforward. Challenge: our algorithm had a tick-tock: side effect, logic, side effect, logic, ... Our approach: separate each side effect and only do one at a time. (16:10) Concept 6: Watch out for convenient side effects Eg. Doing a little bit of I/O in the middle of a function. Eg. Get "now" from the system clock in the middle of a function (16:35) Concept 7: Side-effect assumptions make your system fragile. When you have code in one place that depends on something having been established by code in another place. Eg. Knowing the calling function has already done a check. There is an implicit umbrella of context. Creates a hidden dependency between those two places. When the assumption is violated, there's nothing to stop the code making the assumption from just marching off the cliff. Put the guard logic next to the code that needs it. (19:15) Concept 8: Make context explicit A function should take all of its "knowledge" in as data. It should not assume something has been done for it elsewhere. "This forces you to write down all the things that matter instead of assuming something else has put things in place." Explicit context allows you to separate out imperative steps. "The hidden relationships, the ones that are only in my brain, those are the things that will completely burn me in 6 months." (20:45) Concept 9: Favor instructions as data rather than instructions as function names. Eg. Our Twitter wrapper only has one invoke function. (Inspired by aws-api.) Function names aren't part of the data. If the "command" name is the function name, how do you match on it? You can't. When commands = function names, you have implicit information: "I must have command X because I'm on line Y." Observation: the description of the goal is usually more stable than the steps to achieve the goal. If all of the instructions are in data, the processing component can completely change how it achieves the goal without the calling component changing how it interacts. (26:00) Concept 10: Treat components within your application as loosely coupled services. What is a "microservice"? When you make a request (aka "data"), ship it out, and get a response back (aka "data"). In Clojure, we can treat component interactions like loosely coupled APIs too: send them rich data, get rich data back. Get loose coupling on the inside of the service, not just between services. (27:25) We want to help programmer effectiveness and reduce burnout. "We want to help you avoid situations where you're beating your head against the wall wondering where things went wrong." "One of the great benefits of simple design and Clojure, it allows you create things that are fun to create AND fun to maintain." Message Queue discussion: (29:15) "You know what else is a lot of fun? Hearing from our listeners!" We had an issue with our SSL certs. Thanks for the heads up and let us know if you have any tech trouble. "Automagically". We can't say we invented it, but we love that word. "What is magic? An invisible force with a visible effect. Sound an awful lot like a side effect!" "You start wielding the power, it feels GREAT! And then it all starts going wrong." Easy does not imply simple. Go watch that talk. (See below.) Related episodes: 020: Data Dessert 021: Mutate the Internet 022: Evidence of Attempted Posting 023: Poster Child 024: You Are Here, But Why? 025: Fake Results, Real Speed 026: One Call to Rule Them All Clojure in this episode: nil Related links: Cognitect aws-api Simple Made Easy…

F

Functional Design in Clojure

1
Ep 026: One Call to Rule Them All 29:57

۶ years پیش29:57

29:57

Christoph thinks goals are data, not function names. We were talking about the twitter handle again. Last week, we talked about faking. It's not mocking. The magic that makes it possible is using a protocol. Switch out the real handle or the fake handle in component based on configuration. "Yes, I do want to speak to the log file." "Sometimes, the log file gets lonely." (02:43) Christoph wants to talk about the protocol. We made a function for each operation we did in Twitter. Post tweet Search Get timeline Each function took the information needed for that operation as individual parameters. (04:32) Example: the AWS API wrapper that Cognitect released. Other wrappers are enormous, with functions for each AWS functions. Cognitect's wrapper has just one: invoke . At least only one that gets work done. Unconventional (aka "weird"). Take a step back. What needs to happen to call a remote API? We need to make an HTTP call to some end point. "100% of the information that that server needs to get from us, we transmit as data." "The path, the headers, everything about it is data." "The function is in the URL and the URL is in the data. It's all data." If you want the operations to be exposed as functions, you end up promoting the operation to be a function name, but when making the request you need to put the operation back into the data. Benefit (not really): we get to write a lot of boilerplate code. "Nothing like some boilerplate to get you warmed up in the morning." (09:15) Back to the Twitter handle, what if we just had one function? "Making a function for each endpoint goes against what we talked about last week, which is making our context explicit." (09:55) First benefit of one function: Simplifies the worker half of our algorithm. Before: Decider creates map with {:command :twitter/fetch-timeline :twitter/last-tweet-id 1234} Worker uses a multi-method to convert that to (handle/fetch-timeline handle {:twitter/last-tweet-id 1234}) Handle function will construct Twitter request {:url "https://twitter.com/fetch-timeline" :last-tweet 1234} Handle sends request to Twitter and handles the response. After: Decider creates map with {:command :twitter/operation :twitter/command :fetch-timeline :twitter/last-tweet-id 1234} Worker's multi-method detects a Twitter operation and passes the entire operation map to the handle. Handle transforms the operation into the data needed for the request. Handle sends request to Twitter and handles the response. When the data is in the map, it's easier to test. (18:03) Second benefit of one function: a place for common code in the handle. Inside the handle, you can use a multi-method. If there is common code, like auth checking or data transformation, that can go into the single function. The same tick-tock that was used to get separate side effects from logic in our algorithm can be used inside the handle. "Imperative logic is like a branching tree of side-effects." The handle works on a rich map of information, and with one function, we're handing it one rich map of information. "You have deciders all the way down." (20:52) How do we do spec this single function? One function needs to take data in multiple shapes. Making a spec that matches all of those shapes will be difficult. This function call other, more specific functions to get its work done, so there can be specific specs on those. "You don't have to spec everything at all the levels." We can also lightly check the entry point to check for attributes that are common to all shapes. "Spec is an open system. I can check for the presence of what I need without having to assert a closed world." (22:55) Thinking about the symmetry. The following are equivalent: 4 functions that each take 5 parameters. 4 functions that each take a map with 5 keys. 1 function that takes a map with 6 keys (the last one being the operation). Programming systems that are strongly typed push you toward more functions and less data. Trampoline from data to function to data to function to data... Clojure's structural checking makes it so that we can have assurances about data without being forced to check or name everything. "Say what you mean, don't make me read your body language and guess." Message Queue discussion: (25:50) Separating logic from side effects. "The tree of side effects is like a bowl where you swirl around into the bottom and then swirl back up to the top." In a call stack, side-effects should be shallow on the stack. "Pure things can't screw you over." "Keep your friends close and your side-effects closer." Running this concept out to its logical conclusion enabled us to discover the tick-tock approach. Related episodes: 006: All Wrapped Up in Twitter 021: Mutate the Internet 022: Evidence of Attempted Posting 023: Poster Child 024: You Are Here, But Why? 025: Fake Results, Real Speed Clojure in this episode: defprotocol defrecord Related links: Cognitect aws-api Stuart Halloway: Running with Scissors…

F

Functional Design in Clojure

1
Ep 025: Fake Results, Real Speed 25:16

۶ years پیش25:16

25:16

Nate wants to experiment with the UI, but Twitter keeps getting the results. "This thing that we're making because we're lazy has now taken 4 or 5 weeks to implement." Last week: "worker" logic vs "decider" logic. Allowed us to flatten the logic. "You can spend months on the backend and people think you've barely got anything done. You can spend two days on the UI and then people think you're making progress." (04:20) Now we want to work on the UI, but we don't want data to post to real Twitter. "Why do you keep posting 'asdf asdf asdf' to Twitter?" UI development involves a lot of exploration. Need to try things out with data. We want to be able to control the data we're working with. "We want carefully-curated abnormal data to test the edges of our UI." (06:30) We could have a second Twitter account to fill with junk data. Or, we could use the Twitter API sandbox. Problem: we don't have much control over the data set. Eg. We can't just "reset" back to what it used to be. Problem: what about when we're hacking on a plane? Plus, we want to be able to share test data between developers. (09:10) What can we do instead? Let's make a "fake" Twitter service we run on our machine. "Fake" Twitter gives us something our application can talk to that is under our control. Having a "fake" Twitter service creates new problems Project wants to grow larger and larger Now we have to run more things Creates more obstacles between checking out the code and getting work done (12:35) Rather than a "fake" Twitter service, we want to "fake it" inside the application. What is "faking it"? Is this the same as "mocking"? Use a protocol for our Twitter API "handle". Allows for an alternative implementation. Not the same as mocking We are not trying to recreate all the possibilities. We are not trying to use it to test the component code. The purpose is different than mocking. The Twitter "faker" is to help us work on the rest of the application. The "faker" is about being productive, not about testing. (17:20) Can have the faker start with useful data. Default data in the faker can launch you straight into dev productivity. "You want automation to support your human-centric exploration of things." "If your environment is as fast as it can possibly be, then it allows you to be as fast as you can possibly be." (20:10) Can use the faker for changing the interaction Eg. have a "slow" mode that makes all the requests take longer. Useful to answer the question: "What does this UI feel like when everything gets slow?" "The faker can implement behavior you need, for exploring the space you need to cover, to converge on the right solution." (22:00) Can have the faker pretend something was posted manually. Allows you to see how the UI behaves when the backend discovers a manual post. (23:00) Goals of faking Support exploration. Not about testing and validation Support the human, creative side of development Support the developer experience Related episodes: 006: All Wrapped Up in Twitter 021: Mutate the Internet 022: Evidence of Attempted Posting 023: Poster Child 024: You Are Here, But Why? Clojure in this episode: defprotocol defrecord component/system-map Related projects: Component…

F

Functional Design in Clojure

1
Ep 024: You Are Here, but Why? 34:37

۶ years پیش34:37

34:37

Christoph needs to test his logic, but he must pry it from the clutches of side effects. Last week we ended up with "imperative mud": lots of nested I/O and logic. (01:45) Difficult to test when all the I/O and logic are mixed together. Object-oriented programming advocates for "mocking" objects with side-effects Mocking reinforces and expands the complexity Complexity stems from all the different branches of execution: mainline, exceptions, corner cases, etc. With the imperative approach, no way to just "jump" to step 2 and test that in isolation. Have to setup all the mocks to tilt the flow of control in the desired direction. The Go language handles the "error flow" problem by making error returns explicit (no exceptions), but that further compounds the branching problem. (05:25) A different approach to deal with nesting: "happy path" route Mainline of execution is the most common case. Any error is thrown which has to be handled out of band. "Exceptions are much better at mocking you than you are at mocking them." Even more problems with mocking: suppose you want to test exponential backoff API handle mock has to fail a couple of times and then succeed. Requires a "stateful" mock! "You have whole object hierarchies that are only in use for your tests!" (07:35) Key observation: imperative programming uses the location of execution as a way of encoding information. Code in the first branch of the "if" statement "knows" something different than code in the second branch. Position in the flow of control is implicit information Change implicit information to explicit information and you can flatten out the code. Implicit: "You are here because you did X." Explicit: "Separating where I am from what to do next." "In an imperative flow, the fact you made it to line 3 means something." (09:55) To get out of the problem of nesting, we need to have positive, represented information instead of implicit, contextual information. Pattern is: logic, I/O, logic, I/O, logic, I/O, etc. Capture each one as a multimethod: a "worker" method and a "decider" method. The "worker" method dispatches on a "command", just does one thing, and returns the result. "My job is not to question why. My job is but to do or die." "I don't know if we need to get into the class struggle." The "decider" method takes a command and a result, looks at them, decides what to do next, returns a command. (18:50) You can work out all the scenarios, and the logic for each scenario is pure! Testing becomes a matter of setting the data Steps can be tested independently (19:45) A "result" isn't simply spewing back the raw data from the I/O Worker picks out the relevant parts and return those Use spec or schema to document responses Can use meta or a nested key (like :raw ) to attach raw data for debugging or the "attempt" log (see Ep 022) Logic should operate on well-known fields, but recording raw responses is useful for telling a story later on. (22:45) Main flow of execution becomes a simple loop + recur Dispatch command to worker and bind response Dispatch command + response to decider and bind new-command (when new-command (recur new-command)) (24:30) A downside of the approach: it can be harder to see cases that are not covered, like an unsupported command or unhandled response Once again, unit tests can help ensure coverage for the "deciders". (27:30) How to test the worker? They don't have any logic except doing the side effect. Even "picking" the right values from the data could be factored as a pure function Can test with the REPL. Try out the side effect and then leave it alone. (29:50) Application testing creates another problem What about testing the UI? We want to see changes in the data, but we don't want to actually post for real. Great topic for next time. Message Queue discussion: (32:05) Do we post the source code for episodes? We post some code in the show notes, but less so for a high-level series. Send us links to your code and we'll put them on the website. Related episodes: 021: Mutate the Internet 022: Evidence of Attempted Posting 023: Poster Child Clojure in this episode: loop recur meta when…

F

Functional Design in Clojure

1
Ep 023: Poster Child 31:20

۶ years پیش31:20

31:20

Nate gets messy finding ingredients for his algorithm cake. Last week we focused on how to determine what to post. This week we focus on getting the data so we can compare it. (01:55) Once again, we'll use component to organize our app. What components should we have? Component 1: The worker that wakes up, checks the DB, checks Twitter, and posts as necessary. Debate: Do we need more than one component? Question: What does that worker need? Those should be components. (03:00) Component 2: The database connection. Needs to be threadsafe Allows all the DB logic in one place. start method is a natural place to do migrations, indexing, etc. (04:45) Aside: quick component refresher Very lightweight mechanism for shared, stateful resources. Dependency injection: components depend on other components, but framework instantiates them. Each component has two lifecycle methods start and stop . Setup state in start : initialize internal data, open connection, prep work, etc. Tear it all down in stop : finishing work, closing connections, etc. Declare all your components in a system . (06:00) What should we call Component 1? "Poster"? "It could be the poster child of components!" (06:50) Component 3: The Twitter API handle What is a "Twitter API handle" (aka the "Twitter handle")? Data structure and code that handles authenticating and making requests. Not a long standing socket connect (like the DB), but there is still state (auth key). (08:50) What happens when the cached credentials expire? Two options from Ep 006: Request function returns a tuple: [updated-handle, result] . The handle will only change when it has to re-auth. Use an atom for the handle. Request function mutates the handle when it has to re-auth. In both cases, the Twitter wrapper deals with re-auth. The difference is whether the code that uses the handle needs to know about it. We'll use #2 for our Twitter component. We must still consider a race condition between components who all use an expired handle at the same time. Extra work will be done, but the result will still be correct. If components use separate handles, they can't benefit from sharing the re-auth. (12:55) We need a way to trigger "waking up" Use at-at to schedule an interval for calling a function Backed by Java's ScheduledThreadPoolExecutor . "It's good to have your components be tidy and clean up after themselves." Important to call at-at/stop in your component/stop method so component.repl/reset doesn't make more and more timers! (16:10) We have the ingredients to implement the algorithm Basic steps: Wrap it all in an exception handler Use a sequence in a let block to assign results back from Twitter and DB Take results and pass them to pure function to determine what to do next Complication: order dependent. We need last posted Tweet to know how far back to fetch from Twitter New steps: Fetch from DB: last posted and next scheduled Use last posted ID to fetch from Twitter Pass tweets and next scheduled tweet to pure function to determine if we should post If we need to post, try posting to Twitter and capture result If success, write tweet ID into database to mark as "completed" No matter what, write the result to the "attempt" log (22:00) Feels very imperative "Do I/O. Do some logic. Do I/O. Do some logic. All those little bits of logic are very hard to test." OO answer: mock the resources Mocking makes more problems: now you have to implement all sorts of fake logic "You just start grabbing more and more side effects and glomming them on to this big ball of mud, just so you can test a little bit of logic." "Next thing you know, you're developing a whole vocabulary of mock creation." We'll look at this more next week. Message Queue discussion: (25:00) We were mentioned on the Illegal Argument podcast Comparing Clojure REPL and Smalltalk REPL. Common problem: state in the REPL does not reflect what is in the source. For us, connected editor helps avoid that: we run pieces directly from our source files. Still can end up out of sync: dangling symbol references. Using tools.namespace.repl/refresh will find those dangling references. Can still build up in pieces, but can use refresh to check it all at once. Related episodes: Twitter handle, retrying on failure 006: All Wrapped Up in Twitter Related projects: Component component.repl At-At Java ScheduledThreadPoolExecutor tools.namespace Clojure in this episode: atom component/ start stop system-map component.repl/ reset at-at/ mk-pool every stop…

F

Functional Design in Clojure

1
Ep 022: Evidence of Attempted Posting 33:52

۶ years پیش33:52

33:52

Christoph questions his attempts to post to Twitter. This week, continuing to dig into the "Twitter problem". We want to post to Twitter on a schedule. "Writing code to help out with laziness." Start with data to keep track of: inside (our data) and outside (Twitter data) "Data from a foreign land." We need to determine our "working view" of Twitter's data. What is in our data? For each "scheduled tweet": Text to post: the "status" Timestamp of when to post Timestamps are nice Milliseconds since the epoch Universal instant Allows the client to localize How do we know a scheduled tweet has been posted? A "posted?" boolean? Boolean says, "Yes! It has been posted somewhere on the Internet." Correlating identifiers are more useful than a Boolean. The tweet ID is a correlating identifier. We can use it to lookup all of Twitter's data about it. "We don't need to store all of Twitter in our database." What is the story you need to tell about what happened? A record of all the attempts allows us to tell a story about what happened. Useful to have the timestamp of when our application posted it. Make a separate log for attempts. Attempting to post is a separate concern than what to post. Don't complicate the scheduled tweet information by embedding the log. "Once you have all the data, it allows you to ask new questions you didn't originally think of." Clojure makes it easy to work with a large tree of data that came from an external source. We don't have to care about the structure of that data. We can just write it down. Simply attempt to post the next scheduled tweet that does not have a Twitter ID recorded. If it fails, just record the attempt, and go back to sleep. "Handle the brick in front of you, and if you keep doing that, you'll eventually build the wall." What if we don't hear the success response from Twitter, but it did get posted? Idea: Try to detect if a tweet has already been posted. If we can uniquely identify something by its content, we can know two things are the same without having a common ID. Problem: Twitter can alter the contents. Idea: fuzzy "measure of similarity" between our recent tweets and the next scheduled tweet. We can record the fuzzy match in our attempt log too! If we can correlate by contents, we could even identify when we manually post in advance. As soon as you can determine equality by the substance of the thing itself, you can have more than one writer. How "recent" is "recent"? Is it 100? Is it 200? Is it 500? Even better, fetch all the tweets since the last ID we recorded. we know we're seeing all of the tweets can scan each of those for a match (in the case of a manual post) know when the tweet stream ends, so we can know a posting is still needed The worker will get there eventually. Can just give up on an error. No complex retry and recovery logic. With more than one writer, we still can have a race condition. Ultimately Twitter has to deal with deduplication to avoid a double post in a short interval. Message Queue discussion: Namespacing in a map is really useful A flat, namespaced map is easier to traverse than a nested map. One use for namespaces: indicate the origin of the data Eg. :twitter/id, :twitter/status vs :local/id, :local/text You see the namespace in your code, so it makes the data origin very visible. Related episodes: Schemas for data. "Internal" vs "external" data. 007: Input Overflow Creating a "big bag of data" and asking it questions. 018: Did I Work Late on Tuesday? 019: Dazed by Weak Weeks 020: Data Dessert Related projects: XKCD: Is It Worth the Time? The rsync algorithm Clojure in this episode: pr-str…

F

Functional Design in Clojure

1
Ep 021: Mutate the Internet 22:28

۶ years پیش22:28

22:28

Nate wants to tweet regularly, so he asks Clojure for some help. Problem: pre-author tweets so they can get posted automatically. Want to make a "full stack" application this time. Sounds complicated, what do we need? Database of tweets: text to tweet and timestamp when it should be tweeted. Frontend is a single-page application (SPA) that makes XHR ("AJAX") requests to the backend. Needs to be able to wake up and post. Persistent process backend. We will not cover all these parts. "You have a problem, so you make a UI to solve your problem. Now you have 2 problems." "More like 18 problems!" We will focus on logic interacting with Twitter. When should it post? How does it know if a tweet has been posted? What to do when Twitter returns an error? Overarching theme: how do you deal with side-effects in a functional way? Remember Ep 020: push side-effects and I/O to the edges. Easy to fetch "current" time, but that's a side effect! "Just because it's easy doesn't mean it's pure." If you make time a parameter, all of a sudden you can mess with it! "If only real time was a parameter we could manipulate." Lots of fun to be had in the upcoming episodes. "'Start with the data' is something we've come to again and again. If you can model the data, that's a very good place to start." Related episodes: 006: All Wrapped Up in Twitter 020: Data Dessert Related projects: Maria Clojure in this episode: nil…

F

Functional Design in Clojure

1
Ep 020: Data Dessert 28:40

۶ years پیش28:40

28:40

Christoph and Nate discuss the flavor of pure data. "The reduction of the good stuff." "We filter the points and reduce the good ones." Concept 1: To use the power of Clojure core, you give it functions as the "vocabulary" to describe your data. "predicate" function: produce truth values about your data "view" or "extractor" function: returns a subset or calculated value from your data "mapper" function: transforms your data into different data "reduction" (or "reducer") function: combines your data together Concept 2: Don't ignore the linguistic aspect of how you name your functions. Reading the code can describe what it is doing. Good naming is for humans. Clojure doesn't care. Concept 3: Transform the data source into a big "bag" data that is true to structure and information of the source. Source data describe the source information well and is not concerned with the processing aspects. Transform into data that is useful for processing. Concept 4: Using loop + recur for data transform is a code smell. Not composable: encourages shoving everything together in one place. "End up with a ball of mud instead of a bag of data you can sift through." "You know what mud sticks to really well? More mud! It's very cohesive! And what couldn't be better than cohesive programs!" Concept 5: Use loop + recur for recursion or blocking operations (like core.async ) Data shows up asynchronously Useful when logic is more naturally expressed as recursion than filter + map + reduce . Concept 6: Duality: stepwise vs aggregate Stepwise problem: advance a game state, apply async event, stream processing, etc. Stepwise: reduce , loop + recur Aggregate problem: selecting the right data and combining it together. Aggregate: filter + map + reduce Aggregate problems tend to be eager--they want to process the whole data set. Concept 7: Use your bag of granular data to work toward a bag of higher-level data. We went from lines → entries → days → weeks "Each level of data allows you to answer different questions." Concept 8: Duality: higher-level data vs granular data with lots of dimensions Eg. having a single "day" record vs a bunch of "entry" records that all share the same "date" field. The "right" choice depends on your usage pattern. Dimensional data tends to stay flat, but high-level data tends toward nesting. A high-level record is a pre-calculated answer you can use over and over quickly. Highly-dimensional, granular record allows you to "ask" questions spanning arbitrary dimensions. Eg. "What weeknights in January did I work past midnight?" Concept 9: Keep it pure. Avoid side effects as much as possible. Pure functions are the bedrock of functional programming. REPL and unit test friendly. "You can use data without hidden attachments. You remember side effects when you're writing them, but you don't remember them three months later." Concept 10: Keep I/O at the "edges" with pure functions in the "middle". "I/O should be performed by functions that you didn't write." Use pure functions to format your data so you only have to hand it off to the I/O function. Eg. Create a list of "line" strings to emit with (run! println lines) . You can describe your I/O operations in data and make a "boring" function that just follows them. This allows you to unit test the complicated logic that determines the operations. Separates out I/O specific problems from business logic problem: eg. retries, I/O exceptions, etc. Related episodes: 002: Tic-Tac-Toe, State in a Row 012: Fiddle with the REPL 018: Did I Work Late on Tuesday? 019: Dazed by Weak Weeks Clojure in this episode: filter , map , reduce loop , recur group-by run! println…

F

Functional Design in Clojure

1
Ep 019: Dazed by Weak Weeks 29:56

۶ years پیش29:56

29:56

Nate wants to see more than data structures in a REPL. Goal: see a text-based calendar of hours worked by day and week. "With the power of the predicate, I summon thee from the bag of data!" If data items share structure, you only need one predicate function, not separate functions per "type". Common pattern: filter then map , then reduce Output: Looks like a calendar with one line per week Show daily total for each day Show weekly total Problem: how do we split the days into weeks? "I don't remember split-weeks in the Clojure cheat sheet." "What day does your week start on? My week starts on Tuesday." "I always assume that when a programmer says 'most', they mean 'most of the people around me'." "I actually think it just means 'me'." "Anecdata: when you have two anecdotes, it makes data." Create higher-level summaries: entry → day → week Sift through data at the level of your question. Problem: A date identifies a "day", but what identifies a week? A "week ID", so to speak. It would be nice if that week ID was ordered, so we could sort on it. Idea: Use the date of the first day of the week uniq, sortable, spans years weeks that start on Tuesday will never have the same ID as weeks that start on Sunday Might as well have the starting day of the week in the map too: :sunday , :tuesday , etc. Convenient to have two different views for the same data in the same data structure immutable data won't change and get inconsistent. it's pre-calculated if you need to use it a lot vs just using a predicate "on the fly". Want to group-by the week: need a predicate from "date ID" → "week ID": (week-id starting-day-of-week day) need to take a "date" and produce "date of first day in week" Eg. (week-id :sunday {:date "2019-03-08" ...}) => "2019-03-03" "You're teaching Clojure core how to describe your data by giving it vocabulary." Write something that converts data to data, not data to println . Principle: move the I/O to the edges make the println a trivial step at the end even making the strings is a data transform everything prior to the println could be tested, even the formatting! "Test it to the level of detail you can tolerate." Much easier to reason about pure functions. Related episodes: 015: Finding the Time 016: When 8 - 1 = 6 017: Data, at Your Service 018: Did I Work Late on Tuesday? Clojure in this episode: filter , map , reduce group-by sort-by with partition-by take-while , drop-while , and split-with Code sample from this episode: ( ns time.week-05 ( :require [ time.week-04 :as week-04 ] [ java-time :as jt ] )) ; Date helpers ( defn start-of-week "Get the the starting date of the week containing local-date. The week will start on the named day. Eg. :sunday" [ starting-day-of-week local-date ] ( jt/adjust local-date :previous-or-same-day-of-week ( jt/day-of-week starting-day-of-week ))) ( defn week-dates "Create a week's worth of dates starting from the given date." [ starting-date ] ( ->> ( range 0 7 ) ( map #( jt/plus starting-date ( jt/days % ))))) ; Aggregates ( defn sum-minutes "Sums the minutes for all kinds: entry, day, week" [ entries ] ( ->> entries ( map :minutes ) ( reduce + ))) ; Conversions ( defn entries->days "Convert a seq of entries to days." [ entries ] ( ->> ( group-by :date entries ) ( map ( fn [[ date xs ]] { :date date :minutes ( sum-minutes xs )})))) ( defn days->week "Convert a seq of days into a week. Week is picked by first date." [ starting-day-of-week days ] ( let [ lookup ( into {} ( map ( juxt :date identity ) days )) starting-date ( start-of-week starting-day-of-week ( :date ( first days ))) all-days ( ->> ( week-dates starting-date ) ( map #( or ( get lookup % ) { :date % :minutes 0 })))] { :starting-day-of-week starting-day-of-week :date starting-date :days ( vec all-days ) :minutes ( sum-minutes all-days )})) ( defn partition-weeks [ starting-day-of-week days ] ( ->> days ( sort-by :date ) ( partition-by #( start-of-week starting-day-of-week ( :date % ))))) ( defn days->weeks "Convert a seq of days into an ordered seq of weeks." [ starting-day-of-week days ] ( ->> ( partition-weeks starting-day-of-week days ) ( map ( partial days->week starting-day-of-week )))) ( comment ( ->> ( week-04/log-times "time-log.txt" ) ( entries->days )) ( ->> ( week-04/log-times "time-log.txt" ) ( entries->days ) ( days->weeks :sunday )) )…

F

Functional Design in Clojure

1
Ep 018: Did I Work Late on Tuesday? 28:51

۶ years پیش28:51

28:51

Christoph wants to teach filter some vocabulary. Continuing our discussion of rummaging through the big bag of data. Mental shift for solving the problem: Prior thinking: one function that has to look at all entries New thinking: filter out irrelevant entries then reduce on just those New mentality emphasizes the problem of "picking" over "combining". Once you have the right set of entries, the reduce becomes trivial. Idea: build up a "vocabulary" of picking. "You build up the language to the level you want to use it at." A "predicate" is simply a function that gives you a truth value. We want to create a set of predicates to use with filter . By convention, predicates in Clojure end with ? . Eg. some? , contains? , every? First predicate to create: (spans-midnight? start-timestamp end-timestamp) Problem: using it is verbose: (filter #(spans-midnight? (:start %) (:end %)) entries) Better idea: have the predicate take an entry. The predicate should speak at the level of the items for filter . Just take entry: (spans-midnight? entry) Usage: (filter spans-midnight? entries) New question: how many minutes did I work on the weekend? New predicate: (weekend? entry) Usage: (filter weekend? entries) "My time in Clojure makes me look at big, long functions and wonder if they should be broken into smaller pieces." Simplify implementation of weekend? with a simpler predicate: (day-of-week? weekday entry) Order matters: put weekday first for partial . Now the weekend? function is a simple or of calls to day-of-week? Even better: use an "extractor" function (day-of-week entry) that returns the day. Useful for day-of-week? but also for any other logic that needs to pull out the day. An "extractor" provides a useful view of the data. Now a weekday? predicate becomes trivial: (not (weekend? entry)) Key idea: the use of language mirrors how we talk about it. Not just about decomposition, but about how it reads linguistically. Can make a predicate for any day of the week with: (partial day-of-week? :sunday) , etc. Use like so: (filter (partial day-of-week? :sunday) entries) "Partial to parameterize a predicate." (Say that three times fast.) New question: did I work a long day on Tuesday? Won't work to write a predicate at the "entry" level Need a new "day" level Once again, the language hints at the level of abstraction. Idea: function that "uplevels" by taking a list of entries and producing a list of days Predicates can work at both levels if entry and day have some consistent structure. The "structure" (or "data shape") is a consistent use of keys and key paths between abstractions. It is not a "base class". Eg.: both entry and day have a :date key, so the same day-of-week? predicate works on both. Related episodes: 015: Finding the Time 016: When 8 - 1 = 6 017: Data, at Your Service Clojure in this episode: filter reduce partial or Code sample from this episode: ( ns time.week-04 ( :require [ time.week-03 :as week-03 ] [ java-time :as jt ])) ; Helper for loading up time entries ( defn log-times [ filename ] ( ->> ( week-03/lines filename ) ( week-03/times ))) ; Extractors ( defn day-of-week [ entry ] ( jt/day-of-week ( -> entry :date ))) ; Predicates ( defn spans-midnight? [ entry ] ( not= ( jt/local-date ( :start entry )) ( jt/local-date ( :end entry )))) ( defn day-of-week? [ day entry ] ( = ( day-of-week entry ) ( jt/day-of-week day ))) ( defn weekend? [ entry ] ( or ( day-of-week? :saturday entry ) ( day-of-week? :sunday entry ))) ( defn weekday? [ entry ] ( not ( weekend? entry ))) ; Aggregations ( defn total-minutes [ entries ] ( ->> entries ( map :minutes ) ( reduce + ))) ( comment ( ->> ( log-times "time-log.txt" ) ( filter spans-midnight? )) ( ->> ( log-times "time-log.txt" ) ( filter ( partial day-of-week? :wednesday ))) ( ->> ( log-times "time-log.txt" ) ( filter weekend? )) ( ->> ( log-times "time-log.txt" ) ( filter weekday? ) ( total-minutes )) )…

F

Functional Design in Clojure

1
Ep 017: Data, at Your Service 28:28

۶ years پیش28:28

28:28

Nate finds it easier to get a broad view without a microscope. After last week's diversion into time math, we are back to the core problem this week. Now we want a total by date. Need to refactor the function to return the date in addition to minutes. "We're letting the code grow up into the problem." "Let's let the problem pull the code out of us." First attempt Use map to track running totals by day As each new entry is encountered, update the total for that day in the map New complication: Now we want a total for all work on Sundays. The loop + recur approach is getting complicated! More and more concerns all mixed together in one place Closely ties the traversal of the data to the processing of the data Better idea: use reduce . Just write "reducer" functions. Simplify by ensuring data passed to reduce is already filtered. "In imperative land, let's take three different dimensions of consideration and shove them all together in this one zone." Motivating question for a solution: "How is this composable?" "In Clojure you end up with really small functions because you end up composing them at the end." Ugly: the reducer for "work on Sundays" still has an if for throwing away data. Better: add another filter to just pass through Sundays. Best: minimal work in the reducer. Use map and filter to get the data in shape first. Imperative thinking: what value do I need to operate on? Functional thinking: how can I accurately represent the data present in the input? After you have all the data at hand, you can summarize it however you want! Why reducers? When you need to operate one step at a time: streaming data, game state, etc. Clojure's sequence abstraction is powerful and unifying. "All the functions in the core work on all the data." Related episodes: 015: Finding the Time 016: When 8 - 1 = 6 Clojure in this episode: loop , recur map , filter , reduce group-by if -> , ->> Code sample from this episode: (ns time.week-03 (:require [clojure.java.io :as io] [clojure.string :as string] [java-time :as jt])) ; Functions for parsing out the time format: Fri Feb 08 2019 11:30-13:45 (def timestamp-re #"(\w+\s\w+\s\d+\s\d+)\s+(\d{2}:\d{2})-(\d{2}:\d{2})") (defn localize [dt tm] (jt/zoned-date-time dt tm (jt/zone-id))) (defn parse-time [time-str] (jt/local-time "HH:mm" time-str)) (defn parse-date [date-str] (jt/local-date "EEE MMM dd yyyy" date-str)) (defn adjust-for-midnight [start end] (if (jt/before? end start) (jt/plus end (jt/days 1)) end)) (defn parse [line] (when-let [[whole dt start end] (re-matches timestamp-re line)] (let [date (parse-date dt) start (localize date (parse-time start)) end (adjust-for-midnight start (localize date (parse-time end)))] {:date date :start start :end end :minutes (jt/time-between start end :minutes)}))) ; How many minutes did I work on each day? (defn daily-total-minutes [times] (->> times (group-by :date) (map (fn [[date entries]] (vector date (reduce + (map :minutes entries))))) (into {}))) ; How many minutes total did I work on Sundays? (defn on-sunday? [{:keys [date]}] (= (jt/day-of-week date) (jt/day-of-week :sunday))) (defn sunday-minutes [times] (->> times (filter on-sunday?) (map :minutes) (reduce +))) ; Functions for turning the time log into a sequence of time entries (defn lines [filename] (->> (slurp filename) (string/split-lines))) (defn times [lines] (->> lines (map parse) (filter some?))) ; Process a time log with the desired summary calculation (defn summarize [filename calc] (->> (lines filename) (times) (calc))) (comment (summarize "time-log.txt" daily-total-minutes) (summarize "time-log.txt" sunday-minutes) )…

F

Functional Design in Clojure

1
Ep 016: When 8 - 1 = 6 29:37

۶ years پیش29:37

29:37

Christoph discovers that time creates its own alternate universe. Continuing our exploration of "literate" time logs We want a function to turn the timestamps into minutes. Format: Fri Feb 08 2019 11:30-13:45 Keep it simple: extract times with a regex and use math minutes in day = hours * 60 + minutes total minutes = end minutes - starting minutes Problem: What happens when we work past midnight? Negative minutes! We decided to have only one date, so a time before the starting time must span midnight. Format only allows for an activity to be <24 hours. "If we end up doing any activity in our life longer than 24 hours, I think we should that we might have other problems." Easy Solution: If the end time is before start time, add 24 hours. "When I get past any sort of simpler math, I just type it into my connected editor REPL because I know Clojure can do it faster than I can." Now we have a function to get minutes, want to add them all up. Use loop and recur to iterate through the array and track the sum. Oh wait, what about Daylight Savings Time? "We all pretend that time is actually in the future." If it involves dates and times, we can't just do simple math. "If I do this the right way, I now have to open a whole new can of worms." Easy way out: write "doesn't support DST" in the release notes and call it "user error"! "Any time you have to be careful about something, you're probably doing it wrong." Use a time API. "The Clojure library is just a wrapper because nobody wants to reinvent this thing." Java time is immutable, so that works nicely with functional languages. No clone ing! Lots of functions in the time API. Which ones do we need? Our workflow: try out different expressions in a comment block in our connected editor to figure out the specific time functions we need. "Local" dates and times don't have time zones, but "zoned" dates and times do have them. Need to create timestamps for accurate math: date + time + zone In practice, when we use dates and times, they are implicitly connected to a time zone. Your time zone is your alternate universe: it affects the meaning of your dates and times. We added support for DST without changing the function signature. But, how do we add up other totals? Looks like we're going to need to change even more. Related episodes: 015: Finding the Time Clojure in this episode: nil re-matches loop , recur Related projects: Java Time Joda Time Code sample from this episode: (ns app.time (:require [clojure.java.io :as io] [java-time :as jt])) (def timestamp-re #"(\w+\s\w+\s\d+\s\d+)\s+(\d{2}:\d{2})-(\d{2}:\d{2})") (defn localize [dt tm] (jt/zoned-date-time dt tm (jt/zone-id "America/Los_Angeles"))) (defn parse-time [time-str] (jt/local-time "HH:mm" time-str)) (defn parse-date [date-str] (jt/local-date "EEE MMM dd yyyy" date-str)) (defn parse-for-minutes [line] (if-let [[whole dt start end] (re-matches timestamp-re line)] (let [date (parse-date dt) start (localize date (parse-time start)) end (localize date (parse-time end))] (if (jt/before? start end) (jt/time-between start end :minutes) (jt/time-between start (jt/plus end (jt/days 1)) :minutes))) 0)) (defn total-time [filename] (with-open [rdr (io/reader filename)] (loop [total 0 lines (line-seq rdr)] (if lines (recur (+ total (or (some-> (first lines) parse-for-minutes) 0)) (next lines)) total))))…

F

Functional Design in Clojure

1
Ep 015: Finding the Time 26:22

۶ years پیش26:22

26:22

Nate spends some time figuring out how to track his time. New problem: Want to track time using a text file. Text files are great: Keyboard oriented: in terminal, in editor Something you can put under revision control Need: date and time range Format of the time stamp should be convenient for us to read and edit. Let the computer do the work to convert the convenient notation to something usable! Format: Fri Feb 08 2019 11:30-13:45 One timestamp per line. Oh wait, we need a description. Six minutes in and we're already changing the requirements! What are "literate" text files? "You can mix human words in amongst data the computer will use to make it more understandable for the humans." How to find the times? Attempt to match the time format within each line. "It's kind of like an inverse comment. Instead of every line being valid and you have to comment out lines you don't want, if it's in a known format, those are the lines we want and everything else is a comment." Can use line-seq with clojure.java.io/reader . That uses lazy I/O. Potential Issue: lazy I/O can defer I/O errors to when you're in the middle of processing the data. Lazy I/O is less of an issue with files and more of an issue with network sockets. Another approach: slurp in all the data at once and call split-lines "Trade a little memory for some safety." Clojure uses the seq abstraction whichever way you choose. It's Clojure's unifying abstraction. "Even maps get squeezed into a seq if you push them hard enough." Next step: figure out which lines are time entry lines and which lines are commentary Use a regex to match against each string using Clojure's built-in #"..." form. In other situations, you can use instaparse for grammar-based parsing. Put timestamps on their own line Easier to work with Allows for more error detection (in the future) Use capture groups and re-matches detect the match and extract the parts. Use when-let to destructure and do something only if it matches Better yet, make a function time-info that does the match and returns either nil or the string that matched. For now, just start by using doseq to just go through all the lines and test our matching. Making sense of the timestamp string is a whole new problem for next time. Clojure in this episode: nil seq line-seq clojure.java.io/reader slurp clojure.string/split-lines #"" , re-matches let , when-let doseq Related projects: Instaparse Code sample from this episode: (def timestamp-re #"(\w+)\s(\w+)\s(\d+)\s(\d+)\s+(\d{2}):(\d{2})-(\d{2}):(\d{2})") (with-open [rdr (clojure.java.io/reader "time-log.txt")] (doseq [line (line-seq rdr)] (when-let [[whole d m dt yr hs ms he me] (re-matches timestamp-re line)] (println whole))))…

F

Functional Design in Clojure

1
Episode 014: Fiddle With the REPL 29:28

۶ years پیش29:28

29:28

Christoph gets some work done by fiddling around. Editor integration is a massive change in how to use the REPL. Put a comment block right underneath a function you are working on, and invoke the function with test arguments. Can quickly make changes to code and try it out--all in the same place! Problem: comment blocks littering the code. "A good rule of thumb I've found is: when I have to use the world 'careful', it's a big red flag. I'm doing something I should be able to prevent without having to be careful." Code in comment blocks can be helpful in the future. "'It's not done yet." What file is ever done? You're always going to come back to it in 6 months." Easy to skip around between forms you've run before. "It's like a random access REPL history." Ah ha moment: make a separate file for the helpful comment blocks. "[The comment blocks] are like the copilot of my development experience." "They're me helping me!" "You're your own copilot!" Separate file needed a namespace: the fiddle namespace was born. "I'm just fiddling around." Put the "fiddle files" in the dev tree so it only gets loading in development. "These fiddle files started multiplying, like little helpful rabbits." Like pages in a notebook centered on a theme. Each "page" is a file: a file for remembering how to call certain core functions a file for experimenting with date and time conversions a file for exploring a new concept a file for interacting with new code being developed Never have to leave to editor. REPL is working behind the scenes. Unlike REPL history, you can share fiddle files. Fiddles can help show the developer's mindset during development. "It's like seeing into the back of your mind of the way you explore this on your own." Example: exploring a data set work on code for parsing a data file write different expressions to sift through the data e.g. threaded expression to: open the data, parse it, filter , and map build up helpers for making sense of the data parser and helpers in fiddle are the start of the "real" application code write examples with commentary Example: exploring the working state of the application set of utility functions for grabbing information from component that set of helpers can be share between developers code that pulls out state from specific components all at once "Where did this nil come from?" Example: exploring external APIs write simple function for calling the HTTP endpoint start calling it with different examples can easily see and re-run calls that you've tried Slow API? Use (def response (...)) to save a result, and write expressions using response . "A fiddle file like a scratch pad for new features." Characteristics of fiddles a working set of expressions like a notebook with code around a theme like a scratch pad for exploring toward a solution can be shared written context from solving the problem "The me of the future won't have as much access to the mental state of the me of the present." Do your future self a favor and write down some of your current mental context. Could make educational fiddles for new developers in a project. "Fiddles are a way of capturing process: a way of capturing the craft of making it, not just the outcome." Related episodes: 012: Embrace the REPL 013: Connect the REPL 002: Tic-Tac-Toe, State in a Row Clojure in this episode: comment def , defn -> , ->> filter , map , reduce select-keys clojure.pprint/ pprint print-table Related projects: Editors with REPL integration Project Jupyter Component Joda Time Java Time…

94 subscribers

مشابه Functional Design in Clojure

Medicube Zero Pore Pads 2.0, Dual-Textured Facial Toner Pads for Exfoliation and Pore Care with 4.5% AHA Lactic Acid & 0.45% BHA Salicylic Acid, Ideal for All Skin Types, Korean Skin Care (70 units)

Elmer's All Purpose School Glue Sticks, Acid-Free and Washable, Clear, 7 Grams, 30 Count - Ideal for Craft Projects, Scrapbooking, Classroom, Home, Office, Teacher Supplies

TERRO Ant Killer Bait Stations T300B - Liquid Bait to Eliminate Ants - 12 Count Stations for Effective Indoor Ant Control

پادکست هایی که ارزش شنیدن دارند

Functional Design in Clojure « » Ep 078: Impartial Thoughts

Ep 078: Impartial Thoughts

پادکست هایی که ارزش شنیدن دارند

Functional Design in Clojure « »
Ep 078: Impartial Thoughts