• ES Español
• EN English

5.53. Big Data (Mandatory)

5.53.1. Justification ↑ Back to top

In the current era, the volume, velocity, and variety of data exceed the capabilities of traditional storage and processing systems. This course introduces the fundamental concepts and technologies of Big Data. It covers distributed file systems, parallel processing frameworks, NoSQL databases, and real-time data streaming. Students will learn to design scalable architectures to extract value from massive datasets.

5.53.2. Generales Goals ↑ Back to top

1. Understand the 5 V's of Big Data and their impact on system design.
2. Design and implement scalable solutions using the MapReduce paradigm.
3. Master the use of distributed file systems like HDFS.
4. Analyze different architectural patterns for Big Data (Lambda and Kappa).
5. Apply stream processing techniques for real-time data analysis.

5.53.3. Contribution to Outcomes ↑ Back to top

AG-C11) Use of Tools: Applies modern computing tools in problem solving. (Usage)

AG-C09) Design and Development of Solutions: Designs, implements, and evaluates solutions for complex computing problems. (Usage)

5.53.4. Content ↑ Back to top

5.53.4.1. Core Database System Concepts (12 hours) [Skills AG-C09,AG-C11] ↑ Back to top

Bibliography: (White, 2015; Karau et al., 2017)

Topics

1. Purpose and advantages of database systems
2. Components of database systems
3. Design of core DBMS functions (e.g., query mechanisms, transaction management, buffer management, access methods)
4. Database architecture, data independence, and data abstraction
5. Transaction management
6. Normalization
7. Approaches for managing large volumes of data (e.g., NoSQL database systems, use of MapReduce)
8. How to support CRUD-only applications
9. Distributed databases/cloud-based systems
10. Structured, semi-structured, and unstructured data
11. Use of a declarative query language
12. Systems supporting structured and/or stream content

Learning Outcomes

1. Identify at least four advantages that using a database system provides [Analyze]
2. Enumerate the components of a (relational) database system [Enumerate]
3. Follow a query as it is processed by the components of a (relational) database system [Analyze]
4. Defend the value of data independence [Defend]
5. Compose a simple select-project-join query in SQL [Compose]
6. Enumerate the four properties of a correct transaction manager [Enumerate]
7. Describe the advantages for eliminating duplicate repeated data [Describe]
8. Outline how MapReduce uses parallelism to process data efficiently [Outline]
9. Evaluate the differences between structured and semi/unstructured databases [Evaluate]

5.53.4.2. Distributed Databases/Cloud Computing (12 hours) [Skills AG-C09,AG-C11] ↑ Back to top

Bibliography: (White, 2015; Lin and Dyer, 2010)

Topics

1. Distributed DBMS:
2. Distributed data storage
3. Distributed query processing
4. Distributed transaction model
5. Homogeneous and heterogeneous solutions
6. Client-server distributed databases enumerate
7. Parallel DBMS:
8. Parallel DBMS architectures: shared memory, shared disk, shared nothing
9. Speedup and scale-up, e.g., use of the MapReduce processing model
10. Data replication and weak consistency models enumerate

Learning Outcomes

1. Describe the key components of a distributed DBMS, including distributed data storage, query processing, and transaction management [Describe]
2. Analyze the tradeoffs between parallel DBMS architectures: shared memory, shared disk, and shared nothing [Analyze]
3. Describe data replication strategies and weak consistency models in distributed database systems [Describe]

5.53.4.3. NoSQL Systems (12 hours) [Skills AG-C09,AG-C11] ↑ Back to top

Bibliography: (Sadalage and Fowler, 2012; Kleppmann, 2017b)

Topics

1. Why NoSQL? (e.g., Impedance mismatch between Application [CRUD] and RDBMS)
2. Key-Value and Document data model
3. Storage systems (e.g., Key-Value systems, Data Lakes)
4. Distribution Models (Sharding and Replication)
5. Graph Databases
6. Consistency Models (Update and Read, Quorum consistency, CAP theorem)
7. Processing model (e.g., Map-Reduce, multi-stage map-reduce, incremental map-reduce)
8. Case Studies: Cloud storage system (e.g., S3); Graph databases; "When not to use NoSQL"

Learning Outcomes

1. Develop a use case for the use of NoSQL over RDBMS [Create]
2. Describe the defining characteristics behind Key-Value and Document-based data models [Describe]

5.53.4.4. Data Analytics (12 hours) [Skills AG-C09,AG-C11] ↑ Back to top

Bibliography: (Matei et al., 2012; Bill and Matei, 2018; Kleppmann, 2017b; Psaltis, 2017)

Topics

1. Exploratory data techniques (motivation, representation, descriptive statistics, visualizations)
2. Data science lifecycle: business understanding, data understanding, data preparation, modeling, evaluation, deployment, and user acceptance
3. Data mining and machine learning algorithms: e.g., classification, clustering, association, regression
4. Data acquisition and governance
5. Data security and privacy considerations
6. Data fairness and bias
7. Data visualization techniques and their use in data analytics
8. Entity Resolution

Learning Outcomes

1. Describe several data exploration approaches, including visualization, to understanding unfamiliar datasets [Describe]
2. Apply several data exploration approaches to understanding unfamiliar datasets [Apply]
3. Describe basic machine learning/data mining algorithms and when they are appropriate for use [Describe]
4. Apply several machine learning/data mining algorithms [Apply]
5. Describe legal and ethical considerations in acquiring, using, and modifying datasets [Describe]
6. Describe issues of fairness and bias in data collection and usage [Describe]

5.53.5. Bibliography ↑ Back to top

White, T. (2015). Hadoop: The Definitive Guide. O'Reilly Media, 4th edition.

Karau, H., Konwinski, A., Wendell, P., and Zaharia, M. (2017). Learning Spark: Lightning-Fast Big Data Analysis. O'Reilly Media.

Lin, J. and Dyer, C. (2010). Data-Intensive Text Processing with MapReduce. Morgan and Claypool Publishers.

Sadalage, P. J. and Fowler, M. (2012). NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence. Addison-Wesley.

Kleppmann, M. (2017b). Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems. O'Reilly Media.

Matei, Z., Mosharaf, C., Tathagata, D., Ankur, D., Justin, M., Murphy, M., J, F. M., Scott, S., and Ion, S. (2012). Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In Proceedings of the 9th USENIX Symposium on Networked Systems Design and Implementation (NSDI), San Jose, CA. USENIX Association.

Bill, C. and Matei, Z. (2018). Spark: The Definitive Guide: Big Data Processing Made Simple. O'Reilly Media, Sebastopol, CA.

Psaltis, A. (2017). Streaming Data: Understanding the real-time pipeline. Manning Publications.