Episode 007: Input Overflow

Functional Design in Clojure

Nate just wants to print the tweets, but the input wants to take over.

• Got tweet fetching working last time. Now we want to print out the tweets.
• API returns a lot information
• "We should probably not use pprint as our output mechanism."
• Using the uniqueness filtering from last time
• The goal is NOT to write a generic Twitter wrapper.
• Goal is to write a wrapper that just gives us what our application needs.
• "I don't want to spread the complexity of Twitter all over my app. I want to hide it inside of the wrapper."
• Clojure data structures are generic enough already!
• We pick out just what we need and put it in our own data structure.
• We use our "internal" structure throughout the application code.
• Our internal structure protects our application code from changes in Twitter's format.
• Keep the structure minimal and grow it when the application code needs more fields.
• Where should we put the internal model? Make it a part of the wrapper.
• A wrapper should expose a data model
• A "sequencing" function just threads steps defined by other functions. Pure steps are easy to test, so make the I/O steps as minimal as possible.
• Technique is called "pushing I/O to the edges".
• Let any exceptions fly back to the main loop. It can sleep a little and retry.
• Separate out formatting from println, so you can unit test output.
• Put the cache behind its own data model
• The cache data model provides logical operations like (defn filter-new [cache tweets] ...)
• The filter-new function would return [cache new-tweets]. Cache could have changed.
• How do you know what's in the data model exposed by the wrapper?
• "What is in this thing that we're passing around? You don't just want to read the code and figure it out?"
• "You rapidly get to the point where you don't know what's in your data structure."
• Use schema.
• Put the schemas the wrapper exposes in its own namespace.

Parts of a data model

• the data structure: what that model needs to store or represent
• a schema (or spec) to define and validate the data structure
• "constructor" functions that take args and return a data structure
• "update" functions that take the data structure and return a modified one
• "output" functions that take the data structure and return some other representation

Parts of a wrapper

• Application-facing functions. Sole job is to sequence internal wrapper functions.
• Data model for a Twitter API "request"
• Functions that execute a request and return raw data
• Data model exposed by the wrapper

Clojure in this episode:

• println
• get-in
• ->
• loop, recur, reduce
• lists as "tuples"
• schema/defschema, schema/Int, schema/String, ...

Related projects:

• core.cache
• schema