• ES Español
• EN English

5.53. Software Quality (Elective)

5.53.1. Justification ↑ Back to top

Delivering software that is merely "functional" is no longer sufficient for industry: modern organizations demand systems that are reliable, secure, maintainable, and built through disciplined, auditable engineering processes. This course teaches software quality as a first-class, hands-on engineering discipline practiced through the same tools and workflows used by professional teams on GitHub: branch protection, mandatory code review, automated testing at every level, continuous integration/continuous delivery (CI/CD), static analysis, and production observability.

Students progress from foundational quality models through advanced collaborative-development workflows, automated testing strategies (unit, integration, end-to-end, and behavior-driven), CI/CD pipelines, static analysis and refactoring, and production-quality concerns such as supply-chain security, performance, and reliability engineering. The course closes with an emphasis on DevOps culture – the shared team practices and values (Definition of Done, blameless postmortems, continuous learning) that sustain quality at scale.

5.53.2. Generales Goals ↑ Back to top

1. Evaluate software quality using an internationally recognized quality-characteristics model and reason about the cost of poor quality and technical debt.
2. Apply a professional, review-gated Git/GitHub workflow, including branch protection, CODEOWNERS, and conventional commits.
3. Design and implement unit, integration, end-to-end, and behavior-driven test suites, including test-driven development.
4. Build and operate CI/CD pipelines that enforce automated quality gates, dependency scanning, and semantic versioning.
5. Apply static analysis, mutation testing, and refactoring techniques to identify and remove code smells and technical debt.
6. Analyze production-quality concerns, including performance testing and reliability/observability practices.
7. Adopt DevOps culture practices – Definition of Done, blameless postmortems, and continuous team learning – to sustain quality.

5.53.3. Contribution to Outcomes ↑ Back to top

AG-C09) Design and Development of Solutions: Designs, implements, and evaluates solutions for complex computing problems. (Assessment)

AG-C11) Use of Tools: Applies modern computing tools in problem solving. (Assessment)

AG-C12) Applies computer science theory and software development fundamentals to produce computer-based solutions. (Usage)

5.53.4. Content ↑ Back to top

5.53.4.1. Software Design Quality and Evaluation (4 hours) [Skills AG-C09,AG-C12] ↑ Back to top

Bibliography: (International Organization for Standardization, 2011; Sommerville, 2015)

Topics

1. Data design Data Modeling
   1. Data structures
   2. Storage systems
2. Requirement traceability
   1. Understanding which requirements are satisfied by a design
3. Design modeling, for instance with class diagrams, entity relationship diagrams, or sequence diagrams
4. Measurement and analysis of design quality
5. Principles of secure design and coding Security Design and Controls Engineering , Threat Analysis and Security Engineering , Trusted Computing and Privacy Engineering
   1. Principle of least privilege
   2. Principle of fail-safe defaults
   3. Principle of psychological acceptability
6. Evaluating design tradeoffs (e.g., efficiency vs reliability, security vs usability)
7. ISO/IEC 25010 quality characteristics model and cost of poor quality
   1. Quality characteristics: functional suitability, performance efficiency, compatibility, usability, reliability, security, maintainability, portability
   2. Cost of poor quality and technical debt as a business/economic concern, not just a technical one

Learning Outcomes

1. Design a set of data structures to implement a provided API surface [Design]
2. Identify which requirements are satisfied by a provided software design [Analyze]
3. Translate a natural language software design into class diagrams [Translate]
4. Adapt a flawed system design to better follow the principles of least privilege and fail-safe defaults [Create]
5. Contrast two software designs across different qualities, such as efficiency or usability [Contrast]
6. Evaluate a software system against the ISO/IEC 25010 quality characteristics and estimate the business cost of poor quality/technical debt [Evaluate]

5.53.4.2. Version Control and CI/CD (12 hours) [Skills AG-C11,AG-C12] ↑ Back to top

Bibliography: (Kim et al., 2021)

Topics

1. Software configuration management and version control: Software Development Practices
   1. Configuration in version control, reproducible builds/configuration.
   2. Version control branching strategies. Development branches vs release branches. Trunk-based development.
   3. Merging/rebasing strategies, when relevant.
2. Release management.
3. Testing tools including static and dynamic analysis tools. Software Development Practices , Information Flow and Non-Interference , Injection and Input Validation , Memory Safety and Types , Malware Analysis and Advanced Security
4. Software process automation:
   1. Build systems - the value of fast, hermetic, reproducible builds, compare/contrast approaches to building a project.
   2. Continuous Integration (CI) - the use of automation and automated tests to do preliminary validation that the current head/trunk revision builds and passes (basic) tests.
   3. Continuous Deployment (CD) - the use of automation to automatically release every change that passes the automated tests to the production environment, ensuring frequent and reliable deliveries.
   4. Dependency management - updating external/upstream dependencies, package management, SemVer.
5. Collaborative review workflow as a quality gate
   1. Mandatory pull-request review before merging
   2. Code ownership via CODEOWNERS-style rules
   3. Conventional commit message conventions
   4. Branch-protection rules enforcing checks before merge
6. Dependency and supply-chain vulnerability scanning integrated into CI pipelines

Learning Outcomes

1. Describe the difference between centralized and distributed software configuration management [Describe]
2. Describe how version control can be used to help manage software release management [Describe]
3. Identify configuration items and use a source code control tool in a small team-based project [Analyze]
4. Describe how available static and dynamic test tools can be integrated into the software development environment [Describe]
5. Understand the use of CI/CD systems as a ground-truth for the state of the team's shared code (build and test success) [Explain]
6. Apply a pull-request review workflow with code-ownership rules and branch protection to enforce quality gates on a team project [Apply]
7. Use a dependency/supply-chain vulnerability scanner as part of a CI pipeline to detect vulnerable dependencies [Use]

5.53.4.3. Test Planning and Types (8 hours) [Skills AG-C09] ↑ Back to top

Bibliography: (Sommerville, 2015)

Topics

1. Test kinds
   1. Unit
   2. Integration
   3. Validation
   4. System
2. Stylistic differences between tests and production code: DAMP vs DRY - more duplication is warranted in test code.
3. Test planning and generation
   1. Test case generation, from formal models, specifications, etc.
   2. Test coverage
      1. Test matrices
      2. Code coverage - how much of the code is tested?
      3. Environment coverage - how many hardware architectures, operating systems, browsers, etc. are tested?
   3. Test data and inputs

Learning Outcomes

1. Compare and contrast the different types and levels of testing (regression, unit, integration, systems, and acceptance) [Compare]
2. Describe techniques for creating a test plan and generating test cases [Describe]
3. Create a test plan for a medium-size code segment which includes a test matrix and generation of test data and inputs [Create]
4. Implement a test plan for a medium-size code segment [Implement]

5.53.4.4. Advanced Testing Practices (4 hours) [Skills AG-C11] ↑ Back to top

Bibliography: (Beck, 2002)

Topics

1. Test development
   1. Test-driven development
   2. Object oriented testing, mocking, and dependency injection
   3. Opaque-box (previously, black-box) and transparent-box (previously, white-box) testing techniques
   4. Test tooling, including code coverage, static analysis, and fuzzing
2. Verification and validation in the development cycle
   1. Code reviews
   2. Test automation, including automation of tooling
   3. Pre-commit and post-commit testing
   4. Tradeoffs between test coverage and throughput/latency of testing
   5. Defect tracking and prioritization: reproducibility of reported defects
3. Domain specific verification and validation challenges
   1. Performance testing and benchmarking
   2. Asynchrony, parallelism, and concurrency
   3. Safety-critical
   4. Numeric

Learning Outcomes

1. Identify the fundamental principles of test-driven development methods and explain the role of automated testing in these methods [Analyze]
2. Discuss issues involving the testing of object-oriented software [Debate]
3. Describe mocking and dependency injection and their application [Describe]
4. Undertake, as part of a team activity, a code review of a medium-size code segment [Evaluate]
5. Describe the role that tools can play in the validation of software [Describe]
6. Automate the testing in a small software project [Design]
7. Explain the roles, pros, and cons of pre-commit and post-commit testing [Explain]
8. Discuss the tradeoffs between test coverage and test throughput/latency and how this can impact verification [Debate]
9. Use a defect tracking tool to manage software defects in a small software project [Use]
10. Discuss the limitations of testing in certain domains [Debate]

5.53.4.5. Testing Tools and Analysis (8 hours) [Skills AG-C11] ↑ Back to top

Bibliography: (Wynne and Hellesøy, 2012)

Topics

1. Verification and validation tooling and automation
   1. Static analysis
   2. Code coverage
   3. Fuzzing
   4. Dynamic analysis and fault containment (sanitizers, etc.)
   5. Fault logging and fault tracking
2. Test planning and generation
   1. Fault estimation and testing termination including defect seeding
   2. Use of random and pseudo random numbers in testing
3. Testing asynchronous, parallel, and concurrent systems
4. Verification and validation of non-code artifacts (documentation, training materials)
5. Behavior-driven development (BDD)
   1. Expressing acceptance criteria as Gherkin Given/When/Then scenarios
   2. Executing BDD scenarios as automated end-to-end tests
6. Mutation testing and static code-quality analysis
   1. Mutation testing to assess test-suite effectiveness
   2. Static detection of code smells and excessive cyclomatic complexity

Learning Outcomes

1. Describe and compare different tools for verification and validation [Describe]
2. Automate the use of different tools in a small software project [Design]
3. Explain how and when random numbers should be used in testing [Explain]
4. Describe approaches for fault estimation [Describe]
5. Estimate the number of faults in a small software application based on fault density and fault seeding [Estimate]
6. Describe techniques and issues with testing asynchronous, concurrent, and parallel software [Describe]
7. Create a test plan for a medium-size code segment which contains asynchronous, concurrent, and/or parallel code, including a test matrix and generation of test data and inputs [Create]
8. Describe techniques for the verification and validation of non-code artifacts [Describe]
9. Write Gherkin Given/When/Then scenarios for a feature's acceptance criteria and execute them as automated end-to-end tests [Write]
10. Analyze a test suite's mutation score and refactor tests to kill surviving mutants [Analyze]

5.53.4.6. API Compatibility and Versioning (4 hours) [Skills AG-C11,AG-C12] ↑ Back to top

Bibliography: (Sommerville, 2015)

Topics

1. Hyrum's Law/The Law of Implicit Interfaces
2. Backward compatibility
   1. Compatibility is not a property of a single entity, it's a property of a relationship.
   2. Backward compatibility needs to be evaluated in terms of provider + consumer(s) or with a well-specified model of what forms of compatibility a provider aspires to/promises.
3. Versioning
   1. Semantic Versioning (SemVer)
   2. Trunk-based development

Learning Outcomes

1. Identify both explicit and implicit behavior of an interface and identify potential risks from Hyrum's Law [Analyze]
2. Identify changes that can be broadly considered "backward compatible,'' potentially with explicit statements about what usage is or is not supported [Analyze]
3. Evaluate whether a proposed change is sufficiently safe given the versioning methodology in use for a given project [Evaluate]

5.53.4.7. Refactoring Techniques (4 hours) [Skills AG-C09,AG-C12] ↑ Back to top

Bibliography: (Fowler, 2017)

Topics

1. Refactoring
   1. Standard refactoring patterns (rename, inline, outline, etc.)
   2. Use of refactoring tools in IDE
   3. Application of static-analysis tools (to identify code in need of refactoring, generate changes, etc.)
   4. Value of refactoring as a remedy for technical debt
2. "Large Scale'' Refactoring - techniques when a refactoring change is too large to commit safely (large projects), or when it is impossible to synchronize change between provider + all consumers (multiple repositories, consumers with private code).
   1. Express both old and new APIs so that they can co-exist.
   2. Minimize the size of behavior changes.
   3. Why these techniques are required, (e.g., "API consumers I can see'' vs "consumers I can't see'').

Learning Outcomes

1. Consider inputs from static analysis tools and/or Software Design principles to identify code in need of refactoring [Analyze]
2. Refactor the implementation of an interface to improve design, clarity, etc. with minimal/zero impact on existing users [Create]
3. Plan a complex multi-step refactoring to change default behavior of an API safely [Plan]

5.53.4.8. Performance Testing and Benchmarking (4 hours) [Skills ] ↑ Back to top

Bibliography: (Bondi, 2015; Gregg, 2020)

Topics

1. Performance testing and benchmarking
   1. Throughput and latency
   2. Degradation under load (stress testing, FIFO vs LIFO handling of requests)
   3. Speedup and scaling
      1. Amdahl's law
      2. Gustafson's law
      3. Soft and weak scaling
   4. Identifying and measuring figures of merits
   5. Common performance bottlenecks
      1. Compute-bound
      2. Memory-bandwidth bound
      3. Latency-bound
   6. Statistical methods and best practices for benchmarking
      1. Estimation of uncertainty
      2. Confidence intervals
   7. Analysis and presentation (graphs, etc.)
   8. Timing techniques

Learning Outcomes

1. Describe throughput and latency and provide examples of each [Describe]
2. Explain speedup and the different forms of scaling and how they are computed [Explain]
3. Describe common performance bottlenecks [Describe]
4. Describe statistical methods and best practices for benchmarking software [Describe]
5. Explain techniques for and challenges with measuring time when constructing a benchmark [Explain]
6. Identify the figures of merit, construct and run a benchmark, and statistically analyze and visualize the results for a small software project [Analyze]

5.53.4.9. Reliability Engineering (4 hours) [Skills AG-C11,AG-C12] ↑ Back to top

Bibliography: (Kim et al., 2021; Forsgren et al., 2018)

Topics

1. Software reliability models
2. Software fault tolerance techniques and models
   1. Contextual differences in fault tolerance (e.g., crashing a flight critical system is strongly avoided, crashing a data processing system before corrupt data is written to storage is highly valuable)
3. Software reliability engineering practices - including reviews, testing, practical model checking
4. Identification of dependent and independent failure domains, and their impact on system reliability
5. Measurement-based analysis of software reliability - telemetry, monitoring and alerting, dashboards, release qualification metrics, etc.

Learning Outcomes

1. Demonstrate the ability to apply multiple methods to develop reliability estimates for a software system [Demonstrate]
2. Identify methods that will lead to the realization of a software architecture that achieves a specified level of reliability [Analyze]
3. Identify ways to apply redundancy to achieve fault tolerance [Analyze]
4. Identify single-point-of-failure (SPF) dependencies in a system design [Analyze]

5.53.4.10. Large-Scale Construction and Process (4 hours) [Skills AG-C12] ↑ Back to top

Bibliography: (Forsgren et al., 2018)

Topics

1. Larger-scale testing
   1. Test doubles (stubs, mocks, fakes)
   2. Dependency injection
2. Work sequencing, including dependency identification, milestones, and risk retirement
   1. Dependency identification: Identifying the dependencies between different tasks
   2. Milestones: A collection of tasks that serve as a marker of progress when completed. Ideally, the milestone encompasses a useful unit of functionality.
   3. Risk retirement: Identifying what elements of a project are risky and prioritizing completing tasks that address those risks.
3. Potential security problems in programs Information Flow and Non-Interference , Injection and Input Validation , Memory Safety and Types , Malware Analysis and Advanced Security
   1. Buffer and other types of overflows
   2. Race conditions
   3. Improper initialization, including choice of privileges
   4. Input validation
4. Documentation (autogenerated)
5. Development context: "green field'' vs existing code base
   1. Change impact analysis
   2. Change actualization
6. Release management
7. DevOps practices
   1. Definition of Done as a shared quality bar for completed work
   2. Blameless postmortems for learning from incidents/failures
   3. Team quality culture: shared ownership, feedback, and continuous learning

Learning Outcomes

1. Rewrite a simple program to remove common vulnerabilities, such as buffer overflows, integer overflows and race conditions [Apply]
2. Write a software component that performs some non-trivial task and is resilient to input and run-time errors [Write]

5.53.4.11. Capstone: End-to-End Quality Pipeline Demonstration (8 hours) [Skills AG-C09] ↑ Back to top

Bibliography: (Forsgren et al., 2018)

Topics

1. Integration of all course practices into a single live repository: branch protection, mandatory PR review, automated test suites (unit, integration, E2E/BDD), CI/CD pipeline with quality gates, dependency scanning, static analysis, and observability.
2. Live repository review: demonstration and defense of the team's quality pipeline in front of peers/instructor.
3. Retrospective and blameless postmortem of the project's quality journey.

Learning Outcomes

1. Integrate version control, automated testing, CI/CD, static analysis, and observability into a single, end-to-end quality pipeline for a real repository. [Usage]
2. Defend the design and effectiveness of a team's quality pipeline through a live repository review. [Usage]
3. Conduct a blameless postmortem to identify process improvements for future projects. [Assessment]

5.53.5. Bibliography ↑ Back to top

International Organization for Standardization (2011). Iso/iec 25010:2011 – systems and software engineering – systems and software quality requirements and evaluation (square) – system and software quality models. ISO/IEC.

Sommerville, I. (2015). Software Engineering. Pearson, 10th edition.

Kim, G., Humble, J., Debois, P., Willis, J., and Forsgren, N. (2021). The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations. IT Revolution Press, 2nd edition.

Beck, K. (2002). Test-Driven Development: By Example. Addison-Wesley.

Wynne, M. and Hellesøy, A. (2012). The Cucumber Book: Behaviour-Driven Development for Testers and Developers. Pragmatic Bookshelf.

Fowler, M. (2017). Refactoring: Improving the Design of Existing Code. Addison-Wesley, 2nd edition.

Bondi, A. B. (2015). Foundations of Software and System Performance Engineering: Process, Performance Modeling, Requirements, Testing, Scalability, and Practice. Addison-Wesley, Upper Saddle River, NJ.

Gregg, B. (2020). Systems Performance: Enterprise and the Cloud. Addison-Wesley Professional, Boston, MA, 2nd edition.

Forsgren, N., Humble, J., and Kim, G. (2018). Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations. IT Revolution Press.