Ensimag Rubrique Formation 2022

Large scale Data Management and Distributed Systems - WMM9MO72

  • Number of hours

    • Lectures 30.0
    • Projects -
    • Tutorials -
    • Internship -
    • Laboratory works 6.0
    • Written tests -

    ECTS

    ECTS 6.0

Goal(s)

At the end of the course, the students will have an overview of the challenges associated with storing and processing data at large scale. They will know how to use Big Data software tools to efficiently store and process large amounts of data, including tools that can operate in realtime.

Responsible(s)

Thomas ROPARS

Content(s)

The ability to process large amount of data is key to both industry and research today. As computing systems are getting larger, they generate more data that need to be analyzed to extract knowledge.

Data management infrastructures are growing fast, leading to the creation of large data centers and federations of data centers. Suitable software infrastructures should be used to store and process data in this context. Big Data software systems are build to take advantage of large set of distributed resources to efficiently process massive amounts of data while being able to cope with failures that are frequent at such a scale.

In addition to the amount of data to be processed, the other main challenge that such Big Data systems need to deal with is time. For some use cases, the earlier the results of a data analysis is obtained, the more valuable the result is. Some Big Data systems especially target stream processing where data are processed in realtime.

Through lectures and practical sessions, this course provides an overview of the software systems that are used to store and process data at large scale. The following topics will be covered:

  • Map-Reduce programming model
  • In-memory data processing
  • Stream processing (data movement and processing)
  • Large scale distributed data storage (distributed file systems, NoSQL data bases)

    Throughout the lectures, the challenges associated with performance and fault tolerance will also be discussed.

Prerequisites

Fundamentals of DBMS, concurrent programming (threads)

Test

P = Presentation of research papers and/or graded labs
E = Written exam

N1 = 30% P + 70% E1
N2 = E2

The exam is given in english only FR

Calendar

The course exists in the following branches:

  • Curriculum - Master in Computer Science - Semester 9 (this course is given in english only EN)
see the course schedule for 2022-2023

Additional Information

Course ID : WMM9MO72
Course language(s): FR

You can find this course among all other courses.

Bibliography

Dean, Jeffrey, and Sanjay Ghemawat. "MapReduce: simplified data processing on large clusters." Communications of the ACM 51.1 (2008): 107-113.

Zaharia, Matei, et al. "Apache spark: a unified engine for big data processing." Communications of the ACM 59.11 (2016): 56-65.

Murray, Derek G., et al. "Naiad: a timely dataflow system." Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles. ACM, 2013.

Lakshman, Avinash, and Prashant Malik. "Cassandra: a decentralized structured storage system." ACM SIGOPS Operating Systems Review 44.2 (2010): 35-40.