Data management in large-scale distributed systems - WMM9MO08

Objectif(s)

At the end of the course, the students will have an overview of the challenges associated with storing and processing data at large scale. They will know how to use Big Data software tools to efficiently store and process large amounts of data, including tools that can operate in realtime.

Contenu(s)

The ability to process large amount of data is key to both industry and research today. As computing systems are getting larger, they generate more data that need to be analyzed to extract knowledge.

Data management infrastructures are growing fast, leading to the creation of large data centers and federations of data centers. Suitable software infrastructures should be used to store and process data in this context. Big Data software systems are build to take advantage of large set of distributed resources to efficiently process massive amounts of data while being able to cope with failures that are frequent at such a scale.

In addition to the amount of data to be processed, the other main challenge that such Big Data systems need to deal with is time. For some use cases, the earlier the results of a data analysis is obtained, the more valuable the result is. Some Big Data systems especially target stream processing where data are processed in realtime.

Through lectures and practical sessions, this course provides an overview of the software systems that are used to store and process data at large scale. The following topics will be covered:

• Map-Reduce programming model
• In-memory data processing
• Stream processing (data movement and processing)
• Large scale distributed data storage (distributed file systems, NoSQL data bases)
  
  Throughout the lectures, the challenges associated with performance and fault tolerance will also be discussed.

Fundamentals of DBMS, concurrent programming (threads)