Complex Adaptive Systems Theory, Lean and Scrum Origins 
Over the last few months I have been reading
Adaptive Software Development by
Jim Highsmith. This particular book focuses on the application of
Complex Adaptive Systems theory, or CAS, to the human side of software development. Foundational to the book is the idea that we fail to understand the sharp disparity between orderly systems and complex systems. An orderly system can be extremely complicated but that does not make it complex. Complicated problems can be solved with rigorous optimization whereas complex problems require not only new tools but a mindset of adaptation over optimization and arrival of the fittest over survival of the fittest.
Highsmith aims to help the software development community understand how to harness emergence.
Emergence in complex adaptive systems describes a characteristic in which the interaction of the parts create some greater property or effect that cannot be explained or measured by the individual parts.
John H. Holland, a pioneer of CAS and father of genetic algorithms, defined CAS as:
"A dynamic network of many agents (which may represent cells, species, individuals, firms, nations) acting in parallel, constantly acting and reacting to what the other agents are doing. The control of a CAS tends to be highly dispersed and decentralized. If there is to be any coherent behavior in the system, it has to arise from competition and cooperation among the agents themselves. The overall behavior of the system is the result of a huge number of decisions made every moment by many individual agents."
When I see something described in terms such as "acting and reacting", "dispersed", "decentralized", "cooperation", and "huge number of decisions" I am immediately interested in the implications to organizational behavior and team dynamics. Complex adaptive systems are characterized by a high degree of adaptive capacity which in turn provides resilience to agitation, disruption, and change in environment. The application of CAS theory can be studied in a wide range of contexts as diverse as the stock market, ant colonies, human social groups, and cellular biology.
Jeff Sutherland, one of the founders of
Scrum,
recently posted on his Scrum blog about the background of Scrum and
Lean. In this post he states that "the root of both Scrum and Lean is complex adaptive systems theory" and he goes on by saying "I think Scrum and Lean are complementary implementations of ways to deal with physical reality where things are often not linear, not simple, and not predictable." He underscores other Scrum ideas that fit right in to CAS, such as the idea of Sprints that evolve the code like a biological system. When enough mutation occurs in multiple parts of the organism then a shift occurs to a higher level of functionality. As well, he highlights the fact that management must release control of the team and let it function autonomously as a self-organizing entity that grows the system like a plant.
Along with CAS, Sutherland cites
Goldratt's Theory of Constraints (TOC) and
Brook's subsumption architecture as influencing the Scrum methodology. Lean has many roots from
Deming's work in the early-mid twentieth century. Deming's focus on global optimization and quality control practices are well founded in CAS and constraint theory. Sutherland makes the point that Scrum is built to induce punctuated equilibrium and Highsmith underscores this idea in Adaptive Software Development that the "edge of chaos" is where emergence happens. The term "edge of chaos" describes the turbulent gray area between order and chaos. Dr. Sutherland also highlights that punctuated equilibrium is achieved in Lean through set-based engineering which allows for decisions to be made at the
last responsible moment. This correlates with competition in biological environments through which the most adaptable species wins.
Rod Coffin recently blogged about the implications of a Lean decision making process. He highlighted the
Toyota Product Development System (TPDS) and noted that although the Chief Engineer has ultimate responsibility for a product under development, he exercises subtle control by giving the individual teams great freedom to deliver at regular synchronization points. The team uses set-based engineering to aid in making decisions at the lowest possible level, with vision and arbitration left to the Chief Engineer. Rod also highlights
Alistair Cockburn's observation that unverified decisions, those not incorporated into the system/product and accepted by customer/market, are in-process inventory and ultimately amount to waste in Lean thinking. Ultimately this approach to decision making, with it's potential impact to in-process inventory, can significantly impact throughput (cycle time to deliver new value).
In his post, Sutherland iterates "A thoughtful analysis of the relationship between complex adaptive systems and Lean will show that both Scrum and Lean are instantiations of complex adaptive systems theory… In Scrum, we introduce chaos into the development process and then use an empirical control harness to inspect and adapt a rapidly evolving system." He makes the point that CAS theory can be difficult for the masses to understand and therefore not a good motivator for implementing Scrum. Lean can be difficult to explain as well, however the success of organizations like Toyota provide the
social proof to wake people up and make them listen. Ultimately Lean is just an extra teaching tool in terms of taking Scrum to the next level.
Rod asserts in the above mentioned blog entry that organizations which rely on top-down decision making process are introducing bottlenecks and subsequently disrupting flow and inhibiting collaboration and the self-transcendence of a team. I agree with that assessment and would add that those top-down organizational processes will ultimately lead to a complex system failing to adapt quickly enough and therefore losing out on the emergence that could come from the edge of chaos. Jeff Sutherland wraps up his respective blog entry by summarizing; Scrum is not a child of Lean, use Lean and Toyota to sell Scrum to management audiences, and Lean can be used to help teams understand where a Scrum implementation may be broken.
