Clean and Green
Pragmatic Architecture Patterns
Why?
you build systems
build
understandable
systems
build
understandable, testable
systems
build
understandable, testable, maintainable
systems
build new or transform existing systems
such that they are
understandable, testable, maintainable
How?
Simple is better than complex
The pitch
Most objects are coupled procedures
Methods implicitly depend on
mutable instance state
Methods are therefore
coupled to that state
and therefore to each other
instead…
Build systems around
functional transforms
of simple values and data structures
Objection!
No one argues the
high-level expressivity & convenient testability
of pure functions
So what’s the problem?
>>> objections = {'a'} | {'b'}
“That’s a fine academic toy,
but it can’t build real systems.”
(“real” generally being a euphemism
for “HTML-producing” ;)
“We can’t afford to
rewrite
our whole system!”
These concerns are understandable,
but not true
Claim
You don’t need a full rewrite
(and you definitely should not attempt one)
You can build real systems this way
The pitch
Simple is better than complex
Build systems around
functional transforms
of simple values and data structures
How?
Apply the Clean Architecture
— Uncle Bob Martin
How?
Apply the Clean Architecture
using
Pragmatic Architecture Patterns
Pragmatic
Tools you can apply to existing systems
Techniques for limiting the
required change surface
smaller change surface
↓
iterative incremental improvement
smaller change surface
↓
measurable progress
smaller change surface
↓
higher confidence & likelihood of success
Architecture
Addresses the design of the entire system
Framework for assigning responsibilities
Patterns
Generalized problem types and
solution approaches
The idea is simple
Build systems around
functional transforms
of simple values and data structures
(but it’s not necessarily easy…)
Exhibit A
@expose() @identity.require(identity.has_permission('agreement_delete')) def delete(self, id): agreement = EndUserAgreement.get(id) if agreement.start_date <= date.today(): return {'success': False, 'msg': '<already active msg>'} if EndUserAgreement.query.count() == 1: return {'success': False, 'msg': '<only agreement msg>'} # In order to ensure there are no gaps in agreements, … previous_agreement = self.get_previous(agreement.start_date, id) if previous_agreement: previous_agreement.end_date = agreement.end_date elif agreement.end_date: # If the deleted agreement was the first one, then we find… next_agreement = self.get_next(agreement.start_date, id) if next_agreement: next_agreement.start_date = agreement.start_date agreement.delete() return {'success': True}
Fetch the agreement to delete from the ORM
def delete(self, id): agreement = EndUserAgreement.get(id) # …
Check that it is not yet active
def delete(self, id): # … if agreement.start_date <= date.today(): return {'success': False, 'msg': '<already active msg>'} # …
(and format a message back if it is)
and that it is not the only agreement
def delete(self, id): # … if EndUserAgreement.query.count() == 1: return {'success': False, 'msg': '<only agreement msg>'} # …
def delete(self, id): # … previous_agreement = self.get_previous(agreement.start_date, id) if previous_agreement: previous_agreement.end_date = agreement.end_date elif agreement.end_date: next_agreement = self.get_next(agreement.start_date, id) if next_agreement: next_agreement.start_date = agreement.start_date
Engage
def delete(self, id): # … agreement.delete() return {'success': True}
So what’s the problem?
Q:
Q:
How would you test this?
How would you test
- 5–6 ORM calls
- ≥ 3 business rules
- ≥ 5 axes of responsibility
@expose() @identity.require(identity.has_permission('agreement_delete')) def delete(self, id): agreement = EndUserAgreement.get(id) if agreement.start_date <= date.today(): return {'success': False, 'msg': '<already active msg>'} if EndUserAgreement.query.count() == 1: return {'success': False, 'msg': '<only agreement msg>'} # In order to ensure there are no gaps in agreements, … previous_agreement = self.get_previous(agreement.start_date, id) if previous_agreement: previous_agreement.end_date = agreement.end_date elif agreement.end_date: # If the deleted agreement was the first one, then we find… next_agreement = self.get_next(agreement.start_date, id) if next_agreement: next_agreement.start_date = agreement.start_date agreement.delete() return {'success': True}
Q:
Q:
How would you implement
custom rules
if a client asked?
Counterpoint
How could we possibly convert
delete()
to a purely functional form?
(for Pete’s sake, dan, even the name has state mutation in it!)
Pattern 1: FauxO
Functional core, imperative shell
Imperative shell:
procedural “glue” that offers
an OO interface & manages dependencies
Functional core:
implements all the decisions
Key rule
Never mix decisions and dependencies
logic goes only in the functional core
dependencies go only in the imperative shell
@expose() @identity.require(identity.has_permission('agreement_delete')) def delete(self, id): agreement = EndUserAgreement.get(id) if agreement.start_date <= date.today(): return {'success': False, 'msg': '<already active msg>'} if EndUserAgreement.query.count() == 1: return {'success': False, 'msg': '<only agreement msg>'} # In order to ensure there are no gaps in agreements, … previous_agreement = self.get_previous(agreement.start_date, id) if previous_agreement: previous_agreement.end_date = agreement.end_date elif agreement.end_date: # If the deleted agreement was the first one, then we find… next_agreement = self.get_next(agreement.start_date, id) if next_agreement: next_agreement.start_date = agreement.start_date agreement.delete() return {'success': True}
becomes
@expose() @identity.require(identity.has_permission('agreement_delete')) def delete(self, id): success, msg = agreements.delete(id) return {'success': success, 'msg': msg}
Our HTTP endpoint now does its
one job
call routing
@expose() @identity.require(identity.has_permission('agreement_delete')) def delete(self, id): success, msg = agreements.delete(id) return {'success': success, 'msg': msg}
We’ve reduced its responsibility surface four fold
It no longer has to change with
@expose() @identity.require(identity.has_permission('agreement_delete')) def delete(self, id): success, msg = agreements.delete(id) return {'success': success, 'msg': msg}
agreements is a manager object in the imperative shell
agreements gathers all the dependencies: stateful objects, system settings, required libraries
What does it look like?
Step 1: is_removable()
# agreements.py (imperative shell) def delete(assignment_id): agreement = EndUserAgreement.get(id) all_agreements = EndUserAgreement.query removable, reason = is_removable(agreement, all_agreements) # date adjustments temporariliy elided… if removable: agreement.delete() return removable, reason
Notice the pivot
agreement.delete() is a mutation applied to a persisted (dependent) object
whereas
is_removable() is logic that can be applied to a simple data structure
Build systems around
functional transforms
of simple values and data structures
What do we mean by
simple values and data structures
- atomic types: str, int, …
- structs or records
- collections of same: list, set, dict
Litmus test: is_removable() should work on a plain, non-ORM object
>>> from collections import namedtuple >>> Agreement = namedtuple('Agreement', 'start_date end_date')
# agreements_core.py (functional core) >>> def is_removable(agreement, all_agreements): ... assert agreement and agreement in all_agreements ... ... if agreement.start_date <= date.today(): ... return False, 'already_active' ... elif len(all_agreements) <= 1: ... return False, 'only_agreement' ... else: ... return True, None
>>> from datetime import date >>> only_agreement = Agreement(date.today(), None) >>> removable, status = is_removable(only_agreement, [only_agreement]) >>> removable False
>>> really_planning_ahead = date(3025, 1, 1) >>> current_agreement = Agreement(date.today(), really_planning_ahead) >>> next_agreement = Agreement(really_planning_ahead, None) >>> removable, status = is_removable(next_agreement, [current_agreement, ... next_agreement]) >>> removable True
But don’t we still have decisions in the shell?
# agreements.py (imperative shell) def delete(assignment_id): agreement = EndUserAgreement.get(id) all_agreements = EndUserAgreement.query removable, reason = is_removable(agreement, all_agreements) # date adjustments temporariliy elided… if removable: agreement.delete() return removable, reason
Practicality beats purity
I might just leave this
The decision was made in the core;
the shell is merely acting on that decision
However
That’s a pretty fine distinction…
you can’t always rationalize this way
which leads to
Pattern 2: Callbacks
# agreements.py (step 2) (imperative shell) def delete(assignment_id): agreement = EndUserAgreement.get(id) all_agreements = EndUserAgreement.query removable, reason = is_removable(agreement, all_agreements, remove_callback=agreement.delete) # date adjustments temporariliy elided… return removable, reason
# agreements_core.py (step 2) (functional core) >>> def is_removable(agreement, all_agreements, remove_callback=None): ... assert agreement and agreement in all_agreements ... ... if agreement.start_date <= date.today(): ... return False, 'already_active' ... elif len(all_agreements) <= 1: ... return False, 'only_agreement' ... else: ... remove_callback() if remove_callback else None ... return True, None
Callbacks can help bridge boundary gaps between
lower-level mechanisms (web, db)
and higher level policy layers
without coupling policies to mechanisms
Callbacks are excellent for
limiting
the required change surface
Quick example: what exam types are available to a candidate?
def available_types(all_types, …, check_functions=()): # other checks… return [exam_type for exam_type in all_types if not any(fn(exam_type) for fn in check_functions)]
If any check function returns an error message, the type is unavalable to the candidate.
Example check: organization credit hold
def registration_open(self, exam_type): organization = self.candidate.organization if organization.registration_blocked(self.candidate, exam_type): return 'registration_blocked_org_credit_hold'
When I did this particular refactor, I was working on
applications
Organizations are two subsystems away…
The check function callback allowed me to
circumscribe
how much I needed to change
“I will refactor applications, and no further”
available_types() is still a pure function,
testable with just data
def test_exam_type_available_if_check_is_false(): exam_type = object() check_function = lambda exam_type: None assert exam_type in \ available_types([exam_type], check_functions=[check_function]) def test_exam_type_not_available_if_check_is_true(): exam_type = object() check_function = lambda exam_type: 'I_dont_think_so' assert exam_type not in \ available_types([exam_type], check_functions=[check_function])
Callbacks are a powerful tool
but easy to overuse
Keep calm
and
apply judiciously
So where were we?
# agreements.py (step 2) (imperative shell) def delete(assignment_id): agreement = EndUserAgreement.get(id) all_agreements = EndUserAgreement.query removable, reason = is_removable(agreement, all_agreements, remove_callback=agreement.delete) # date adjustments temporariliy elided… return removable, reason
With the date adjustments
def delete(assignment_id): agreement = EndUserAgreement.get(id) all_agreements = EndUserAgreement.query removable, reason = is_removable(agreement, all_agreements, remove_callback=agreement.delete) # In order to ensure there are no gaps in agreements, … previous_agreement = self.get_previous(agreement.start_date, id) if previous_agreement: previous_agreement.end_date = agreement.end_date elif agreement.end_date: # If the deleted agreement was the first one, then we find… next_agreement = self.get_next(agreement.start_date, id) if next_agreement: next_agreement.start_date = agreement.start_date return removable, reason
Challenge: disentangle the mutation from the rules
Rules
- what should be updated
- how it should be updated
Pattern 3: Delegated value
Shell assigns a value computed by the core
# agreements.py (step 3) (imperative shell) def delete(assignment_id): agreement = EndUserAgreement.get(id) all_agreements = EndUserAgreement.query def on_remove(): agreement.delete() adjust_dates(minimum_start_date=agreement.start_date) removable, reason = is_removable(agreement, all_agreements, remove_callback=on_remove) return removable, reason
# agreements.py (step 3) (imperative shell) def adjust_dates(minimum_start_date=None): all_agreements = EndUserAgreement.query.order_by('start_date') for agreement, start, end in mind_the_gap(all_agreements, minimum_start_date): agreement.start_date = start agreement.end_date = end
Find ordered pairs of agreements with gaps between them…
def adjust_dates(minimum_start_date=None): all_agreements = EndUserAgreement.query.order_by('start_date') for agreement, start, end in mind_the_gap(all_agreements, minimum_start_date): # …
def adjust_dates(minimum_start_date=None): for agreement, start, end in …: agreement.start_date = start agreement.end_date = end
The core implements the rules
- which agreements need to be updated
- what the new dates should be
a little itertools help (from stdlib docs)
>>> from itertools import izip, tee >>> def pairwise(iterable): ... "s -> (s0,s1), (s1,s2), (s2, s3), ..." ... a, b = tee(iterable) ... next(b, None) ... return izip(a, b)
# agreements_core.py (step 3) (functional core) >>> def mind_the_gap(sorted_agreements, minimum_start_date=None): ... first = sorted_agreements[0] ... ... if minimum_start_date and first.start_date > minimum_start_date: ... yield first, minimum_start_date, first.end_date ... ... for a, b in pairwise(sorted_agreements): ... if a.end_date < b.start_date: ... yield a, a.start_date, b.start_date
Stepping back
We started here
@expose() @identity.require(identity.has_permission('agreement_delete')) def delete(self, id): agreement = EndUserAgreement.get(id) if agreement.start_date <= date.today(): return {'success': False, 'msg': '<already active msg>'} if EndUserAgreement.query.count() == 1: return {'success': False, 'msg': '<only agreement msg>'} # In order to ensure there are no gaps in agreements, … previous_agreement = self.get_previous(agreement.start_date, id) if previous_agreement: previous_agreement.end_date = agreement.end_date elif agreement.end_date: # If the deleted agreement was the first one, then we find… next_agreement = self.get_next(agreement.start_date, id) if next_agreement: next_agreement.start_date = agreement.start_date agreement.delete() return {'success': True}
mixed responsibilities
unclear rules
monolithic expression of intent
Practically untestable
Our functional core
>>> def is_removable(agreement, all_agreements, remove_callback=None): ... assert agreement and agreement in all_agreements ... ... if agreement.start_date <= date.today(): ... return False, 'already_active' ... elif len(all_agreements) <= 1: ... return False, 'only_agreement' ... else: ... remove_callback() if remove_callback else None ... return True, None
Functional core (cont.)
>>> def mind_the_gap(sorted_agreements, minimum_start_date=None): ... first = sorted_agreements[0] ... ... if minimum_start_date and first.start_date > minimum_start_date: ... yield first, minimum_start_date, first.end_date ... ... for a, b in pairwise(sorted_agreements): ... if a.end_date < b.start_date: ... yield a, a.start_date, b.start_date
>>> def is_removable(agreement, all_agreements, remove_callback=None): ... assert agreement and agreement in all_agreements ... ... if agreement.start_date <= date.today(): ... return False, 'already_active' ... elif len(all_agreements) <= 1: ... return False, 'only_agreement' ... else: ... remove_callback() if remove_callback else None ... return True, None
Easily testable using simple data structures
- no special setup
- test calls are symmetric with production calls
Clear assignment of responsibilities
- Core → logic
- Shell → dependencies
- Endpoint → dispatch
FauxO interface provides a
familiar façade
to the rest of the system
Our HTTP endpoint
@expose() @identity.require(identity.has_permission('agreement_delete')) def delete(self, id): success, msg = agreements.delete(id) return {'success': success, 'msg': msg}
Callbacks provide
boundaries,
limiting what we’re required to touch
Callbacks allow the core to direct the shell…
without coupling the shell to it
Our imperative shell
def delete(assignment_id): agreement = EndUserAgreement.get(id) all_agreements = EndUserAgreement.query def on_remove(): agreement.delete() adjust_dates(minimum_start_date=agreement.start_date) removable, reason = is_removable(agreement, all_agreements, remove_callback=on_remove) return removable, reason
Imperative shell (cont.)
def adjust_dates(minimum_start_date=None): all_agreements = EndUserAgreement.query.order_by('start_date') for agreement, start, end in mind_the_gap(all_agreements, minimum_start_date): agreement.start_date = start agreement.end_date = end
This example is from a
real system
that serves
real HTML!
No ivory tower constructions here
Is it easy? Perhaps not…
Is it worth it?
Absolutely
T.S. Eliot
Immature poets imitate;
Immature poets imitate;
mature poets steal
—T.S. Eliot
Special thanks to
Brandon Rhodes the Great
from whom I’ve stolen many ideas over the years
Thank you!
♥
@drocco007