Software Development and Delivery and The Story of an Engineer

1. Humble Beginnings and a Desire to Achieve

While he waited for his morning coffee, Achilles’ mind wandered far back into the past, with faint memories of his hometown on the eastern coast of the Mediterranean flashing in front of his eyes.

Achilles graduated from engineering school two decades ago.

What a journey it was! He was now a proud veteran software engineer working for a multinational organisation, the size of which he could only have dreamed of even a few years ago.

The journey was arduous, tortuous, and exhausting, but not more than Achilles’ desire to achieve.

His world two decades ago was homogenous, tiny, and easy to understand. Causes and effects were readily correlated, technical proficiency and higher education were highly regarded, human relationships and networking were never a topic of relevance, and organisational culture was an alien concept. He had no clue of what lay ahead.

Soon after graduation, Achilles accepted an offer from a large corporation for a software programmer position, where he dutifully spent many years.

Although he was trained as an industrial engineer, software programming seemed like a nice extension which, in his view, was within grasp, and no more than a little effort was anticipated to fill the gap. How far from the truth those thoughts were!

His hot black coffee, now sitting comfortably in both hands, Achilles walked down the avenue leading to his modern office. The wind was chilly, and the morning was fresh.

2. The Messy State of Software Development

Achilles often wondered whether the challenges he faced as a software programmer were the product of his behaviour or attitude, whether they were unique to his situation or context, or were just manifestations of a deeper and more pervasive issue in the software industry.

In particular, the following doubts lingered for a long time.

Why is it hard to change, despite the (often unequivocal) evidence and sometimes an existential threat to the organisation’s survival?

How can multiple best practices, methodologies, and assumptions coexist (Agile vs Waterfall, quality vs time vs scope, shareholder profit vs employee satisfaction) with all the apparent paradoxes they carry? Why was it difficult to empirically determine the most suitable course of action?

What is leadership, and how would a consistent view of leadership reconcile the conflicting nature of a leader’s job? Are exceptional leaders scarce, or is the bar too high? Should all staff carry a degree of leadership within them?

Achilles slowly figured out the answers to these questions primarily through serendipity. Still, the latter can only work if the beneficiary is already on a quest to find the truth, armed with an open mind, psychological safety and critical thinking.

Achilles was lucky enough to acquire those three skills while his mind matured, shaped by (sometimes traumatic) experience.

An open mind means giving new ideas a fair go, an enzyme for change. With psychological safety, one can readily admit mistakes, learn from them, and move on without fear of repercussions.

Finally, critical thinking is someone’s ability to pop out of their subjective existence, objectively examine their reality, notice fundamental cracks in the cultural, societal, or value edifice (should the need arise), look for explanations, and pop back in with an action list.

With psychological safety, critical thinking is more accessible as people can safely expose themselves to the truth.

The Role of Emotional Intelligence In Modern Organizations– An Ingredient or Byproduct of Great Leadership?

3. The Newtonian View of the World

The engineering courses that Achilles was taught focused on hard sciences like physics, chemistry, and mathematics, with electrical, mechanical, software, and civil engineering theories built on top.

Exciting and relevant as they may have been, such studies did not explain the entire spectrum of phenomena that Achilles participated in creating during his career.

Engineers work in groups of organisations, constantly interacting with a changing and complex human environment.

The engineer’s perception of the world is that of machine-like regularity. The ability to influence inputs coupled with the absoluteness of nature’s laws meant that the long-term evolution of the human world could be controlled and managed effectively.

The mechanistic mental models that Achilles relied on were deprived of crucial knowledge of human nature and complexity theory and could not explain creativity, faulty judgements (cognitive biases), and what appeared to be the irrational behaviour of his peers. These shortcomings caused Achilles much grief.

The discrepancy between a deterministic view of the world (where the laws of nature are immutable and causality is simple and evident) and a messy, complex one was agonising.

On the one hand, Achilles remained convinced that rational decision-making processes gave him (and his managers) the ability to effectively control the long-term evolution of complex social groups (like teams and organisations).

Therefore, Achilles maintained for a while that there was no need to delve into natural sciences or anything remotely connected with human nature and groups.

First-rate engineers, for instance, tend to take pride in not knowing anything about people. […] Human resource professionals, by contrast, often pride themselves on their ignorance of […] quantitative methods altogether.

— Managing Oneself, Peter F. Drucker, 1999.

As Peter Drucker wrote, there was a feeling of disdain even for anything related to natural or human sciences. In Achilles’ mind, a corollary of that assumption was that relationships were subordinate and inferior to technical proficiency, and teamwork, networking, and organisational culture were irrelevant.

On the other hand, his colleagues refused to play nice, his managers’ decisions seemed erratic, and the world remained unorderly and unpredictable.

But Achilles was lucky, and occasionally through serendipity, other times through research, he pieced together a larger puzzle, a framework with better explanatory powers of how the world functioned.

After that, Achilles’ daily interactions were smoother, and anxiety hit much lower levels.

It was a great relief for Achilles when he finally acknowledged that absolute control of every detail of his world was utterly and desperately illusory.

Explaining Complex Systems in Organisations — An Indispensable Guide for Software Developers

4. Organisational Culture and Group Integration

After the first few weeks in his new job in early 2000, Achilles had made a solid impression of what life in that organisation meant.

Senior members of his team recounted many stories of extraordinary effort, late-night calls with clients, and extra long days of diligent coding to meet critical deadlines. He was often reminded that these criteria were vital if he wished to be successful.

These assumptions on the nature of satisfactory work and achievement puzzled Achilles for a long time. Wasn’t the motto always “work smart, not hard”? Surely his peers were aware of that concept already? Does working late not reflect an endemic state of fire-fighting rather than fire prevention?

Organisational culture manifests at three levels: objects and artefacts, espoused values, and hidden assumptions. Organisational culture specifies what would be an adequate response to challenging situations. The strength of a culture is determined by the anxiety-reduction levels it produces.

Achilles had to wait many years before he came to appreciate the full potential of corporate narratives and organisational culture.

As it turned out, late nights at the office did not always signal faulty processes or a lack of planning. In most cases, it was a way of expressing personal motivation and a genuine commitment to the team and organisation by demonstrating the will to go above and beyond. It was a ritual that heralded complete integration into the team’s culture.

On the other hand, Achilles believed that innovative quality work would eventually pay off. He did not feel comfortable bowing to the mandates of what he saw as arbitrary rules of organisational culture.

These actions resonated with his core values, centred around enjoying time with friends and family, personal growth outside work, and work-life balance.

Achilles made slow progress in this organisation, never felt like a legitimate team member despite his many years there, and eventually moved on to new roles where he could grow.

But the lessons learned were immense, although making sense of them took years longer than expected. Achilles also made lifelong friends whom he called on Christmas and other Middle-eastern high holidays.

Organisational Culture: The Edgar Schein Model

5. Context Shift and New Challenges

A fragile system, including almost all artificial objects (like tea cups or tables) and machines (like laptops and hair dryers), will buckle and break under pressure.

In contrast, Nassim Taleb’s brainchild, an antifragile system (like all living organisms, including nature), thrives under bounded stressors and slowly decays in sedentary lifestyles.

When subjected to minor, localized, but destructive environmental variations, subcomponents of a complex system will break down, fade away, and get replaced with younger, more robust components immunizing the system against future pressures of a similar magnitude.

Although he did not quite understand them in those terms, Achilles interpreted his experiences along similar lines, frequently quoting Nietzsche, “Out of life’s school of war—what doesn’t kill me, makes me stronger”.

During his early years as an immigrant, rife with anxieties, Achilles underwent a series of paradigm shifts, an inner transformation of some sort.

Transformations are always traumatic, especially since, for people with high self-awareness, changes of such fundamental nature and magnitude directly threaten one’s identity and integrity.

Achilles learned to live with multiple identities that didn’t necessarily conflict with or compromise his core beliefs and values system.

Managing and deploying those identities in the relevant context was an important learning exercise. Achilles managed, nevertheless, with the help of his medium-sized library (which looks massive relative to his apartment) and a small but supportive environment.

Archaic ideas and Newtonian-anchored perceptions made way for modern, sophisticated, and complex paradigms better suited to deal with Achilles’ world.

Achilles carries his scars with pride; he is a survivor.

Cultural Transformations and Resistance to Change: Understanding the Risks to Your Organization’s Growth

6. Things Are Not as Simple as They Seem

It’s comical and tragic at the same time when people are compelled to state something as apparent and trivial as “things are not as simple as they seem”. Yet, we often unwittingly assume normality, causality, predictability, agency, and familiarity with every situation we face. In the (modern) world we created, a reductionist approach like this can never work.

But, we are invariably motivated to make these assumptions as they allow us to efficiently make decisions (using cognitive biases or heuristics) that (almost) always work, especially where inconsequential choices are involved. The alternative (engaging rational and slow reasoning processes) is costly and inefficient.

But if things are not always straightforward, what can they be? Natural sciences tell us that systems fall into one of four groups: ordered, random, complicated, and complex. Because we are not always sure which situation we are dealing with, we have a fifth alternative: confusion (see Cynefin framework).

Natural systems abhor randomness as the cost of maintaining it is utterly high. Soon enough, agents within the system will loosely bond and self-organise, and behavioural patterns will emerge. We now have complexity.

Causality, agency, familiarity, and predictability are difficult to obtain in complex systems, but that’s not the problem. The fundamental problem lies in working in a complex system while assuming it’s ordered. Here, we apply strategies that (always!) lead to unintended, second-order consequences.

This website and its ideas on business management and software development are a testament to this author’s journey and the milestones achieved when searching for the fundamental questions of business management, business strategy, and software development.

Self-Organisation in Agile Teams — How and When Does It Work

7. First Principles and Literature Research

The Eastern Mediterranean populations are not famous for their scepticism or questioning of traditions and legacy thoughts (although this was quite the opposite in Greek antiquity and medieval Arabia, where philosophy was born and matured).

Born and bred in the land of figs and olive trees, Achilles was destined to inherit the attributes of the local culture; luckily, it happened only partially. He attributed his sceptical approach to mainstream thinking purely to luck, as the high variance of characteristics is a common and natural phenomenon within a population.

Achilles was rarely satisfied with superficial explanations that did not get to the core of the matter. Nor was he scared to stray away from group thinking to avoid exclusion.

When exclusion was inevitable, he found solace through reading scientific texts and literature research, primarily by returning to the first principles.

Returning to the first research principles involves examining fundamental assumptions and building knowledge from foundational concepts. There are several benefits of this approach:

Clarity: Going back to first principles can help researchers understand the underlying concepts and assumptions that form the basis of their research. This clarity can help ensure that research is grounded in sound principles and that findings are accurate and reliable.
Rigour: By returning to the first principles, researchers can ensure their research is rigorous and comprehensive. This approach requires thoroughly examining the underlying assumptions, theories, and methods used in the study, which can help ensure that the research is methodologically sound.
Innovation: Going back to first principles can also lead to innovation in research. By questioning existing assumptions and approaches, researchers can identify new ways of thinking about a problem or new approaches to solving it.
Generalizability: Research grounded in first principles is more likely to be generalizable to other contexts and settings. By building on fundamental concepts and assumptions, researchers can develop theories and findings that have broader applicability.

Returning to the first principles forces the researcher to examine the fundamental assumptions. For this examination to bear fruit, it must be precise and technical.

When you describe a phenomenon in precise technical terms, you cast aside any emotional biases that might influence your hypothesis around that phenomenon. This technique is instrumental in finding the truth.

8. The Software Engineering Toolkit

8.1 What Is Software Engineering

Software engineering involves designing, developing, and maintaining software using engineering principles and methods. In today’s world, software engineering plays a crucial role in every aspect of our lives, from mobile applications to enterprise systems. In this article, we’ll explore the different aspects of software engineering, including its processes, principles, key concepts, best practices, tools, challenges, and future outlook.

Software Engineering Processes Software engineering processes refer to the methods and frameworks used to design and develop software. Some of the popular software engineering processes include:

Software Development Life Cycle (SDLC) is a process used to design, develop, test, and deploy software. It comprises six stages: requirements gathering, design, development, testing, deployment, and maintenance.
Agile Development Methodologies: Agile development methodologies are iterative and incremental processes prioritising customer satisfaction, teamwork, and flexibility. Scrum and Kanban are some popular agile methodologies.
The Waterfall Model is a linear and sequential approach that comprises five stages: requirements, design, implementation, testing, and maintenance.
DevOps is a software development approach emphasising collaboration, communication, and automation between software development and IT operations teams.

8.2 Software Engineering Principles

Software engineering principles refer to the fundamental concepts and practices that guide the design and development of software. Some of the key software engineering principles include:

Modularization means breaking down a software system into smaller, manageable, independent modules or components. This principle enhances reusability, maintainability, and flexibility.
Abstraction hides complex implementation details from the user and provides a simpler interface. This principle improves usability, reduces complexity, and enhances maintainability.
Encapsulation is the process of hiding the internal workings of a software component and providing only necessary interfaces to the user. This principle enhances security, reduces complexity, and improves maintainability.
Polymorphism refers to the ability of a software component to take multiple forms or shapes. This principle enhances flexibility, reusability, and maintainability.

8.3 Key Software Engineering Concepts

Software engineering concepts refer to the techniques and practices used in software development. Some of the key software engineering concepts include:

Design Patterns: Design patterns are reusable solutions to commonly occurring software problems. Some popular design patterns include singleton, factory, and observer patterns.
Object-Oriented Programming: Object-oriented programming (OOP) is a programming paradigm that uses objects to represent real-world entities. OOP enhances reusability, maintainability, and extensibility.
Test-Driven Development: Test-driven development (TDD) is a software development process emphasising automated tests before writing the actual code. TDD enhances code quality, reduces errors, and improves maintainability.
Continuous Integration (CI) consists of regularly integrating code changes into a central repository and testing them automatically. CI enhances code quality, reduces errors, and improves collaboration.

8.4 Best Practices in Software Engineering

Software engineering best practices refer to the techniques and methods to ensure high-quality and maintainable software. Some of the popular software engineering best practices include:

Coding Standards: Coding standards ensure code consistency, readability, and maintainability. Examples of coding standards include Google’s style guide and the Java coding conventions.
Documentation refers to writing and maintaining clear and concise descriptions of software components and processes. Good documentation enhances usability, reduces errors, and improves maintainability.
Version Control: Version control manages changes to software code and related artifacts over time. Version control systems like Git and SVN enable developers to track changes, collaborate, and roll back changes if necessary.
During code reviews, developers inspect code changes by other developers to ensure quality, consistency, and maintainability. Code reviews can be done manually or using automated tools.

8.5 Challenges in Software Engineering

Despite the benefits of software engineering, there are also several challenges associated with it. Some of the popular challenges include:

Changing Requirements: Requirements can change during software development for various reasons, such as new market trends, user feedback, and changing business needs. Managing changing requirements can be challenging and requires flexibility and adaptability.
Project Management: Software engineering projects can be complex, involving multiple stakeholders, dependencies, and risks. Effective project management requires strong leadership, communication, and collaboration skills.
Technical debt refers to the accumulation of design and implementation issues in software development that can cause problems in the future. Technical debt can be caused by tight deadlines, lack of resources, and changing requirements.
Security is a critical aspect of software engineering, and any security breach can have severe consequences, such as data loss, financial losses, and reputation damage. Ensuring software security requires careful planning, risk analysis, and implementation of appropriate security measures.

8.6 Future Outlook for Software Engineering

Software engineering is dynamic, and several emerging trends are expected to shape its future. Some of the popular trends include:

Artificial Intelligence and Machine Learning: Artificial intelligence (AI) and machine learning (ML) are increasingly used in software engineering to automate tasks, enhance decision-making, and improve software quality.
Cloud Computing: Cloud computing is a rapidly growing trend that allows software applications to be accessed and managed over the internet. Cloud computing offers scalability, flexibility, and cost-effectiveness.
Internet of Things (IoT): The Internet of Things (IoT) is a network of connected devices that can communicate with each other and the internet. IoT offers new opportunities for software engineering, such as smart homes, healthcare, and transportation.
Cybersecurity will continue to be a critical concern in software engineering, and new techniques and tools will be developed to enhance security, such as blockchain technology and biometrics.

9. Final Words

The series of articles the reader will find on our website tackle the age-old and modern challenges software engineers face daily.

The article will cover three broad topics:

Software development: These articles have a technical bend and discuss the current software delivery issues, from project inception to architecture, design, development, testing, and deployment.
Business management: In this series of articles, we take a step back from technical, isolated issues to broader challenges of building and managing teams, delivering projects, and increasing productivity. What the reader will also find is high attention to the complexity associated with human nature and its impact on a business’ evolution and capacity to deliver.
Science and technology: Technology and science shape our world views and influence our culture (and vice versa), and are, therefore, not just interesting but also vital to understand if we were to make sense of our experiences. This series of articles will tackle high-tech subjects like complexity theory and quantum computing.

All this written knowledge aims to help software engineers achieve Operational Excellence, a concept that started with manufacturing but may as well extend to any human endeavour.

We hope you find this information helpful and derive as much joy from reading it as I did writing it.

Principles of Operational Excellence in Software Development