• ES Español
• EN English

5.45. Parallel and Distributed Computing (Mandatory)

5.45.1. Justification ↑ Back to top

With the end of frequency scaling in single-core processors, parallel and distributed computing has become essential for high-performance software development. This course introduces students to the programming models, communication patterns, and coordination mechanisms required to exploit multi-core systems and distributed clusters. Students will learn to design scalable algorithms and evaluate their performance under different computational constraints.

5.45.2. Generales Goals ↑ Back to top

1. Design and implement parallel programs using shared and distributed memory.
2. Master communication and synchronization protocols in distributed systems.
3. Evaluate performance and scalability through formal metrics.
4. Apply parallel algorithms to solve complex computational problems.

5.45.3. Contribution to Outcomes ↑ Back to top

AG-C11) Use of Tools: Applies modern computing tools in problem solving. (Usage)

AG-C09) Design and Development of Solutions: Designs, implements, and evaluates solutions for complex computing problems. (Usage)

5.45.4. Content ↑ Back to top

5.45.4.1. Programs: Declarative Parallelism (4 hours) [Skills AG-C09,AG-C11] ↑ Back to top

Bibliography: (Pacheco and Malensek, 2021)

Topics

1. Parallelism
2. Declarative parallelism: Determining which actions can, or must not, be performed in parallel, at the level of instructions, functions, closures, compound actions, sessions, tasks, and services is the core idea underlying PDC algorithms; failing to do so is the primary source of errors. See also: Algorithms
3. Defining order: for example, using "happens-before" relationships or series-parallel directed acyclic graphs (DAGs) representing programs.
4. Independence: determining when order does not matter, in terms of commutativity, dependencies, preconditions.
5. Guaranteeing order between otherwise parallel actions when necessary, including locks, safe publication; and enforcing communication - sending a message happens before receiving it; and relaxing it when not necessary. enumerate

Learning Outcomes

1. Graphically represent (as a Directed Acyclic Graph - DAG) how to parallelize a compound numeric expression; for example, \(a = (b + c) * (d + e)\) [Design]
2. Explain why consistency and fault tolerance concepts do not arise in purely sequential programs [Explain]

5.45.4.2. Programs: Starting Activities (3 hours) [Skills AG-C09,AG-C11] ↑ Back to top

Bibliography: (Pacheco and Malensek, 2021)

Topics

1. Starting activities
2. Options allowing actions to be performed (eventually) at locations range from hardwired to configuration scripts; also establishing communication and resource management; these are expressed differently in languages and contexts, typically relying on automated provisioning and management by platforms. See also: Resource Management
3. Procedural: Allowing multiple actions to start at a given program point; for example, starting new threads, possibly delimiting their scope or organizing them into hierarchical groups.
4. Reactive: Enabling upon event occurrence by installing an event handler, with less control over when actions start or finish, and may apply even on single processors.
5. Dependent: Enabling upon completion of others; for example, sequencing sets of parallel actions. See also: Coordination .
6. Granularity: The execution cost of action bodies should outweigh the overhead of organizing them. enumerate

Learning Outcomes

1. Write a service that creates a thread (or other form of procedural activation) to return a requested web page to each new client [Write]

5.45.4.3. Programs: Execution Properties (3 hours) [Skills AG-C09,AG-C11] ↑ Back to top

Bibliography: (van Steen and Tanenbaum, 2023)

Topics

1. Execution Properties
2. Non-deterministic execution of actions without order.
3. Consistency: Ensuring agreement among parties on values and predicates when necessary to avoid races, maintain safety and atomicity, or reach consensus.
4. Fault tolerance: Handling failures in parts or communication, including (Byzantine) misbehavior due to unreliable parties and protocols, when necessary to maintain progress or availability. See also: System Reliability .
5. Trade-offs are a focus of evaluation. See also: Evaluation . enumerate

Learning Outcomes

1. Write a function that counts events, such as network packet receptions, efficiently [Write]

5.45.4.4. Programs: Distribution (2 hours) [Skills AG-C09,AG-C11] ↑ Back to top

Bibliography: (van Steen and Tanenbaum, 2023)

Topics

1. Distribution
2. Defining locations as devices that execute actions, including hardware components, remote hosts; may also include external uncontrolled devices, hosts, and users. See also: Interfacing and Communication
3. A device may time-share or emulate multiple parallel actions using fewer processors through scheduling and virtualization. See also: Scheduling
4. Naming or identifying locations (e.g., device IDs) and actions as parts (e.g., thread IDs).
5. Activities across locations may communicate through media. See also: Communication enumerate

Learning Outcomes

1. Write a filter/map/reduce program in multiple styles [Write]

5.45.4.5. Programs: Implementation Mappings (2 hours) [Skills AG-C09,AG-C11] ↑ Back to top

Bibliography: (Pacheco and Malensek, 2021; mei W. Hwu et al., 2022)

Topics

1. One or more of the following mappings and mechanisms across layered systems:
2. Instruction-level and data parallelism in the CPU. See also: Functional Organization .
3. SIMD and heterogeneous data parallelism. See also: Heterogeneous Architectures .
4. Scheduled concurrency on multi-core, tasks, actors. See also: Scheduling .
5. Clusters, clouds; elastic provisioning. See also: Common Aspects: Platforms, APIs and Constraints , Common Aspects: Platform Languages and Web Programming Patterns .
6. Networked distributed systems. See also: Networked Applications .
7. Emerging technologies such as quantum computing and molecular computing. enumerate

Learning Outcomes

1. Explain the trade-offs between different mapping strategies in terms of performance, cost, and implementation complexity [Explain]

5.45.4.6. GPU Programming (4 hours) [Skills AG-C09] ↑ Back to top

Bibliography: (Kirk and mei W. Hwu, 2016; Corporation, 2024)

Topics

1. General-Purpose GPU Computing (GPGPU)
2. GPU architecture overview: streaming multiprocessors (SMs), CUDA cores, and warp execution model.
3. SIMT (Single Instruction, Multiple Threads) execution model and its differences from CPU SIMD.
4. Host-device interaction: data transfer between CPU (host) and GPU (device) memory via PCIe.
5. Use cases: scientific computing, machine learning, image processing, and large-scale simulation. enumerate
6. CUDA Programming Model
7. CUDA execution hierarchy: grids, thread blocks, and individual threads; mapping to GPU hardware.
8. Kernel definition and launch syntax; configuration parameters (grid and block dimensions).
9. Thread indexing: threadIdx, blockIdx, blockDim, gridDim; computing global indices in 1D, 2D, and 3D.
10. Warp divergence: performance implications of branching within a warp.
11. Intra-block synchronization primitive: __syncthreads. enumerate
12. CUDA Memory Hierarchy
13. Global memory: large, high-latency, accessible by all threads; coalesced access patterns for performance.
14. Shared memory: low-latency on-chip memory shared within a thread block; bank conflicts.
15. Registers and local memory: per-thread private storage.
16. Constant and texture memory: read-only caches optimized for specific access patterns.
17. Unified Memory: simplified programming model with automatic host-device data migration. enumerate

Learning Outcomes

1. Write a CUDA kernel that performs a data-parallel operation (e.g., vector addition, matrix multiplication) and correctly configure grid and block dimensions [Write]
2. Analyze the impact of global memory access patterns (coalesced vs. strided) and shared memory usage on GPU kernel performance [Analyze]
3. Design a parallel algorithm using the CUDA programming model, including thread hierarchy, memory allocation strategy, and host-device data transfers [Design]

5.45.4.7. Communication (8 hours) [Skills AG-C09,AG-C11] ↑ Back to top

Bibliography: (van Steen and Tanenbaum, 2023; Kleppmann, 2017b)

Topics

1. Media
2. Varieties: channels (message passing or I/O), shared memory, heterogeneous, data stores
3. Reliance on the availability and nature of underlying hardware, connectivity, and protocols; language support, emulation. See also: Interfacing and Communication enumerate
4. Channels
5. Explicit (usually named) party-to-party communication media
6. APIs: Sockets, architectural, language-based, and toolkit constructs, such as Message Passing Interface (MPI), and layered constructs such as Remote Procedure Call (RPC). See also: Fundamentals of Networks and Communications
7. I/O channel APIs enumerate
8. Memory
9. Shared memory architectures in which parties directly communicate only with memory at given addresses, with extensions to heterogeneous memory supporting multiple memory stores with explicit data transfer across them; for example, GPU local and shared memory, Direct Memory Access (DMA)
10. Memory hierarchies: Multiple layers of sharing domains, scopes, and caches; locality: latency, false-sharing
11. Consistency properties: Bitwise atomicity limits, coherence, local ordering enumerate
12. Data Stores
13. Cooperatively maintained data structures implementing maps and related ADTs
14. Varieties: Owned, shared, sharded, replicated, immutable, versioned enumerate

Learning Outcomes

1. Explain the similarities and differences among: (1) Party A sends a message on channel X with contents 1 received by party B (2) A sets shared variable X to 1, read by B (3) A sets "X=1'' in a distributed shared map accessed by B [Explain]
2. Write a program that distributes different segments of a data set to multiple workers, and collects results (for the simplest example, summing segments of an array) [Write]
3. Write a parallel program that requests data from multiple sites and summarizes them using some form of reduction [Write]
4. Compare the performance of buffered versus unbuffered versions of a producer-consumer program [Compare]

5.45.4.8. Communication: Properties and Extensions (3 hours) [Skills AG-C09,AG-C11] ↑ Back to top

Bibliography: (van Steen and Tanenbaum, 2023; Kleppmann, 2017b)

Topics

1. One or more of the following properties and extensions
2. Topologies: Unicast, Multicast, Mailboxes, Switches; Routing via hardware and software interconnection networks
3. Media concurrency properties: Ordering, consistency, idempotency, overlapping communication with computation
4. Media performance: Latency, bandwidth (throughput) contention (congestion), responsiveness (liveness), reliability (error and drop rates), protocol-based progress (acks, timeouts, mediation)
5. Media security properties: integrity, privacy, authentication, authorization. See also: Information Flow and Non-Interference , Injection and Input Validation , Memory Safety and Types , Malware Analysis and Advanced Security
6. Data formats: Marshaling, validation, encryption, compression
7. Channel policies: Endpoints, sessions, buffering, saturation response (waiting vs dropping), rate control
8. Multiplexing and demultiplexing many relatively slow I/O devices or parties; completion-based and scheduler-based techniques; async-await, select and polling APIs
9. Formalization and analysis of channel communication; for example, CSP
10. Applications of queuing theory to model and predict performance. enumerate

Learning Outcomes

1. Determine whether a given communication scheme provides sufficient security properties for a given usage [Determine]
2. Give an example of a scenario in which blocking message sends can deadlock [Create]
3. Describe at least one design technique for avoiding liveness failures in programs using multiple locks [Describe]

5.45.4.9. Memory and Consistency (3 hours) [Skills AG-C09,AG-C11] ↑ Back to top

Bibliography: (van Steen and Tanenbaum, 2023; Kleppmann, 2017b)

Topics

1. Memory models, data consistency, and fault tolerance
2. Memory models: sequential and release/acquire consistency
3. Memory management; including reclamation of shared data; reference counts and alternatives
4. Bulk data placement and transfer; reducing message traffic and improving locality; overlapping data transfer and computation; impact of data layout such as array-of-structs vs struct-of-arrays
5. Emulating shared memory: distributed shared memory, Remote Direct Memory Access (RDMA)
6. Data store consistency: Atomicity, linearizability, transactionality, coherence, causal ordering, conflict resolution, eventual consistency, blockchains
7. Faults, partitioning, and partial failures; voting; protocols such as Paxos and Raft.
8. Design tradeoffs among consistency, availability, partition (fault) tolerance; impossibility of meeting all at once
9. Security and trust: Byzantine failures, proof of work and alternatives enumerate

Learning Outcomes

1. Give an example of an ordering of accesses among concurrent activities (e.g., program with a data race) that is not sequentially consistent [Create]
2. Write a program that illustrates memory-access or message reordering [Write]
3. Describe the relative merits of optimistic versus conservative concurrency control under different rates of contention among updates [Describe]
4. Give an example of a scenario in which an attempted optimistic update may never complete [Create]
5. Modify a concurrent system to use a more scalable, reliable, or available data store [Create]
6. Using an existing platform supporting replicated data stores, write a program that maintains a key-value mapping even when one or more hosts fail [Write]

5.45.4.10. Coordination (7 hours) [Skills AG-C09,AG-C11] ↑ Back to top

Bibliography: (Herlihy et al., 2020; van Steen and Tanenbaum, 2023)

Topics

1. Dependencies
2. Initiation or progress of one activity may be dependent on other activities, so as to avoid race conditions, ensure termination, or meet other requirements
3. Ensuring progress by avoiding dependency cycles, using monotonic conditions, removing inessential dependencies enumerate
4. Control constructs and design patterns
5. Completion-based: Barriers, joins, including termination control
6. Data-enabled: Queues, producer-consumer designs
7. Condition-based: Polling, retrying, backoffs, helping, suspension, signaling, timeouts
8. Reactive: Enabling and triggering continuations enumerate

Learning Outcomes

1. Show how to ensure that a program correctly terminates when all of a set of concurrent tasks have completed [Design]
2. Write a function that efficiently counts events such as sensor inputs or networking packet receptions [Write]
3. Write a filter/map/reduce program in multiple styles [Write]
4. Write a program in which the termination of one set of parallel actions is followed by another [Write]
5. Write a service that creates a thread (or other procedural form of activation) to return a requested web page to each new client [Write]

5.45.4.11. Coordination: Synchronization and Atomicity (3 hours) [Skills AG-C09,AG-C11] ↑ Back to top

Bibliography: (Herlihy et al., 2020)

Topics

1. Atomicity
2. Atomic instructions, enforced local access orderings
3. Locks and mutual exclusion; lock granularity
4. Using locks in a specific language; maintaining liveness without introducing races
5. Deadlock avoidance: Ordering, coarsening, randomized retries; backoffs, encapsulation via lock managers
6. Common errors: Failing to lock or unlock when necessary, holding locks while invoking unknown operations
7. Avoiding locks: replication, read-only, ownership, and non-blocking constructions enumerate

Learning Outcomes

1. Show how to avoid or repair a race error in a given program [Analyze]

5.45.4.12. Coordination: Advanced Properties (4 hours) [Skills AG-C09,AG-C11] ↑ Back to top

Bibliography: (Herlihy et al., 2020)

Topics

1. One or more of the following properties and extensions
2. Progress properties including lock-free, wait-free, fairness, priority scheduling, interactions with consistency, reliability
3. Performance with respect to contention, granularity, convoying, scaling
4. Non-blocking data structures and algorithms
5. Ownership and resource control
6. Lock variants and alternatives: sequence locks, read-write locks; Read-Copy-Update (RCU), reentrancy; tickets; controlling spinning versus blocking
7. Transaction-based control: Optimistic and conservative
8. Distributed locking: reliability
9. Alternatives to barriers: Clocks; counters, virtual clocks; dataflow and continuations; futures and RPC; consensus-based, gathering results with reducers and collectors
10. Speculation, selection, cancellation; observability and security consequences
11. Resource control using semaphores and condition variables
12. Control flow: Scheduling computations, series-parallel loops with (possibly elected) leaders, pipelines and streams, nested parallelism
13. Exceptions and failures. Handlers, detection, timeouts, fault tolerance, voting enumerate

Learning Outcomes

1. Write a program that speculatively searches for a solution by multiple activities, terminating others when one is found [Write]
2. Write a program in which a numerical exception (such as divide by zero) in one activity causes termination of others [Write]
3. Write a program for multiple parties to agree upon the current time of day; discuss its limitations compared to protocols such as network transfer protocol (NTP) [Write]

5.45.4.13. Evaluation (7 hours) [Skills AG-C09,AG-C11] ↑ Back to top

Bibliography: (Sterling et al., 2024; Pacheco and Malensek, 2021)

Topics

1. Safety and liveness requirements in terms of temporal logic constructs to express "always'' and "eventually'' See also: Parallel and Distributed Computing
2. Identifying, testing for, and repairing violations, including common forms of errors such as failure to ensure necessary ordering (race errors), atomicity (including check-then-act errors), and termination (livelock)
3. Performance requirements metrics for throughput, responsiveness, latency, availability, energy consumption, scalability, resource usage, communication costs, waiting and rate control, fairness; service level agreements. See also: Latency, Cache and Memory Hierarchy , Virtualization and Isolation
4. Performance impact of design and implementation choices, including granularity, overhead, consensus costs, and energy consumption. See also: Sustainable Design and Pervasive Computing , Environmental Footprint of Computing Systems , Systemic Effects and Social Context
5. Estimating scalability limitations, for example using Amdahl's Law or Universal Scalability Law. See also: Events, Tools and Experimentation , Performance Metrics and Benchmarks , Analytical Performance Models

Learning Outcomes

1. Revise a specification to enable parallelism and distribution without violating other essential properties or features [Redesign]
2. Explain how concurrent notions of safety and liveness extend their sequential counterparts [Explain]
3. Specify a set of invariants that must hold at each bulk-parallel step of a computation [Analyze]
4. Write a test program that can reveal a data race error; for example, missing an update when two activities both try to increment a variable [Write]
5. In a given context, explain the extent to which introducing parallelism in an otherwise sequential program would be expected to improve throughput and/or reduce latency, and how it may impact energy efficiency [Explain]
6. Show how scaling and efficiency change for sample problems without and with the assumption of problem size changing with the number of processors; further explain whether and how scalability would change under relaxations of sequential dependencies [Design]

5.45.4.14. Evaluation: Formal Methods (3 hours) [Skills AG-C09,AG-C11] ↑ Back to top

Bibliography: (Sterling et al., 2024; Pacheco and Malensek, 2021)

Topics

1. Formal verification and analysis methods
2. Extensions to formal sequential requirements such as linearizability
3. Protocol, session, and transactional specifications
4. Use of tools such as Unified Modelling Language (UML), Temporal Logic of Actions (TLA), program logics
5. Security analysis: safety and liveness in the presence of hostile or buggy behaviors by other parties; required properties of communication mechanisms (for example lack of cross-layer leakage), input screening, rate limiting. See also: AI and Hardware Security , Access Control and Applications , Security Mindset and Principles
6. Static analysis applied to correctness, throughput, latency, resources, energy. See also: Sustainable Design and Pervasive Computing , Environmental Footprint of Computing Systems , Systemic Effects and Social Context
7. Directed Acyclic Graph (DAG) model analysis of algorithmic efficiency (work, span, critical paths) enumerate

Learning Outcomes

1. Specify and measure behavior when a service is requested by unexpectedly many clients [Analyze]
2. Identify and repair a performance problem due to sequential bottlenecks [Analyze]
3. Empirically compare throughput of two implementations of a common design (perhaps using an existing test harness framework) [Compare]

5.45.4.15. Evaluation: Testing and Measurement (2 hours) [Skills AG-C09,AG-C11] ↑ Back to top

Bibliography: (Sterling et al., 2024; Pacheco and Malensek, 2021)

Topics

1. Testing tools and measurement techniques
2. Testing and debugging; tools such as race detectors, fuzzers, lock dependency checkers, unit/stress/torture tests, visualizations, continuous integration, continuous deployment, and test generators
3. Measuring and comparing throughput, overhead, waiting, contention, communication, data movement, locality, resource usage, behavior in the presence of excessive numbers of events, clients, or threads. See also: Events, Tools and Experimentation , Performance Metrics and Benchmarks , Analytical Performance Models
4. Application domain specific analyses and evaluation techniques enumerate

Learning Outcomes

1. Identify and repair a performance problem due to communication or data latency [Analyze]
2. Identify and repair a performance problem due to resource management overhead [Analyze]
3. Identify and repair a reliability or availability problem [Analyze]

5.45.4.16. Algorithms (4 hours) [Skills AG-C09,AG-C11] ↑ Back to top

Bibliography: (Pacheco and Malensek, 2021; Herlihy et al., 2020)

Topics

1. Expressing and implementing algorithms in given languages and frameworks, to initiate activities (for example threads), use shared memory constructs, and channel, socket, and/or remote procedure call APIs. See also: Parallel and Distributed Computing .
2. Data parallel examples including map/reduce.
3. Using channel, socket, and/or RPC APIs in a given language, with program control for sending (usually procedural) vs receiving. (usually reactive or RPC-based).
4. Using locks, barriers, and/or synchronizers to maintain liveness without introducing races. enumerate

Learning Outcomes

1. Implement a parallel/distributed component based on a known algorithm [Implement]
2. Write a data-parallel program that for example computes the average of an array of numbers [Write]
3. Write a producer-consumer program in which one component generates numbers, and another computes their average. Measure speedups when the numbers are small scalars versus large multi-precision values [Write]

5.45.4.17. Algorithms: Application Domains Survey (4 hours) [Skills AG-C09,AG-C11] ↑ Back to top

Bibliography: (Pacheco and Malensek, 2021; Herlihy et al., 2020)

Topics

1. Survey of common application domains across multicore, reactive, data parallel, cluster, cloud, open distributed systems, and frameworks (with reference to the following table).
2. Multicore: Typical Execution agents: Threads. Typical Communication mechanisms: Shared memory, Atomics, locks. Typical Algorithmic domains: Resource management, data processing. Typical Engineering goals: Throughput, latency, energy.
3. Reactive: Typical Execution agents: Handlers, threads. Typical Communication mechanisms: I/O Channels. Typical Algorithmic domains: Services, real-time. Typical Engineering goals: Latency.
4. Data parallel: Typical Execution agents: GPU, SIMD, accelerators, hybrid. Typical Communication mechanisms: Heterogeneous memory. Typical Algorithmic domains: Linear algebra, graphics, data analysis. Typical Engineering goals: Throughput, energy.
5. Cluster: Typical Execution agents: Managed hosts. Typical Communication mechanisms: Sockets, channels. Typical Algorithmic domains: Simulation, data analysis. Typical Engineering goals: Throughput.
6. Cloud: Typical Execution agents: Provisioned hosts. Typical Communication mechanisms: Service APIs. Typical Algorithmic domains: Web applications. Typical Engineering goals: Scalability.
7. Open distributed: Typical Execution agents: Autonomous hosts. Typical Communication mechanisms: Sockets, Data stores. Typical Algorithmic domains: Fault tolerant data stores and services. Typical Engineering goals: Reliability. enumerate

Learning Outcomes

1. Extend an event-driven sequential program by establishing a new activity in an event handler (for example a new thread in a GUI action handler) [Design]
2. Improve the performance of a sequential component by introducing parallelism and/or distribution [Create]
3. Choose among different parallel/distributed designs for components of a given system [Assess]

5.45.4.18. Algorithms: Algorithmic Domains (4 hours) [Skills AG-C09,AG-C11] ↑ Back to top

Bibliography: (Pacheco and Malensek, 2021; Herlihy et al., 2020)

Topics

1. One of more of the following algorithmic domains. See also: Algorithmic Strategies :
2. Linear algebra: Vector and matrix operations, numerical precision/stability, applications in data analytics and machine learning.
3. Data processing: sorting, searching and retrieval, concurrent data structures.
4. Graphs, search, and combinatorics: Marking, edge-parallelization, bounding, speculation, network-based analytics.
5. Modeling and simulation: differential equations; randomization, N-body problems, genetic algorithms.
6. Computational logic: satisfiability (SAT), concurrent logic programming.
7. Graphics and computational geometry: Transforms, rendering, ray-tracing.
8. Resource management: Allocating, placing, recycling and scheduling processors, memory, channels, and hosts; exclusive vs shared resources; static, dynamic and elastic algorithms; Real-time constraints; Batching, prioritization, partitioning; decentralization via work-stealing and related techniques.
9. Services: Implementing web APIs, electronic currency, transaction systems, multiplayer games. enumerate

Learning Outcomes

1. Design, implement, analyze, and evaluate a component or application for X operating in a given context, where X is in one of the listed domains, for example, a genetic algorithm for factory floor design [Design]
2. Critique the design and implementation of an existing component or application, or one developed by classmates [Critique]
3. Compare the performance and energy efficiency of multiple implementations of a similar design, for example, multicore versus clustered versus GPU [Compare]

5.45.5. Bibliography ↑ Back to top

Pacheco, P. S. and Malensek, M. (2021). An Introduction to Parallel Programming. Morgan Kaufmann, 2nd edition.

van Steen, M. and Tanenbaum, A. S. (2023). Distributed Systems. Maarten van Steen, 4th edition.

mei W. Hwu, W., Kirk, D. B., and Hajj, I. E. (2022). Programming Massively Parallel Processors: A Hands-on Approach. Morgan Kaufmann, 4th edition.

Kirk, D. B. and mei W. Hwu, W. (2016). Programming Massively Parallel Processors: A Hands-on Approach. Morgan Kaufmann, Cambridge, MA, 3rd edition.

Corporation, N. (2024). Cuda c++ programming guide. https://docs.nvidia.com/cuda/cuda-c-programming-guide/. Documentación oficial de NVIDIA.

Kleppmann, M. (2017b). Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems. O'Reilly Media.

Herlihy, M., Shavit, N., Luchangco, V., and Spear, M. (2020). The Art of Multiprocessor Programming. Morgan Kaufmann, 2nd edition.

Sterling, T., Brodowicz, M., and Anderson, M. (2024). High Performance Computing: Modern Systems and Practices. Morgan Kaufmann, 2nd edition.