CS4402B/CS9635B -- Parallel and Distributed Computing

Course Description

The efficient usage of parallel and distributed systems (multi-processors and computer networks) is nowadays an essential task for computer scientists.

This course studies the fundamental aspects of parallel systems and aims at providing an integrated view of the various facets of software development on such systems: hardware architectures, programming languages and models, software development tools, software engineering concepts and design patterns, performance modelling and analysis, experimenting and measuring, application to scientific computing.

Course topics may include but are not limited to: hierarchical memory, cache complexity, multi-core and many-core architectures, fork-join parallelism, scheduling, scalability, GPU computing, data parallelism, pipelining, message passing (MPI), parallel and distributed data-structures, and applications of parallel and distributed computing.

Follow this link for various resources (software tools and tutorials, hardware documentation, conferences, other HPC course web sites, etc.) regarding this course and HPC in general.

Prerequisites for undergraduate students

• Students must be fluent in C/C++ ; they must also be familiar with UNIX software tools (shell scripts, makefiles, debuggers).
• Students are expected to have background knowledge in computer architecture (e.g. CS3350) and theoretical computer science (e.g. CS3331).

Instructor

Lecture Notes and Textbook

Notes of each lecture will be available on the OWL and course websites.

The following textbooks are recommended but not required:

1. Structured Parallel Programming by Michael McCool, Arch Robison, James Reinder.
2. C++ Concurrency in Action, practical multithreading by Anthony Williams
3. Programming language pragmatics (Chapter 12 only) by Michael L. Scott
4. Models of Computation Exploring the Power of Computing by John E. Savage.
5. CUDA by example by JASON SANDERS and EDWARD KANDROT.

Course and OWL Websites

The course web site is accessible here

For CS4402, the OWL web site is here.

Please check both the course and OWL sites often for updates on lecture notes and errata, as well as the forum discussions and announcements in the OWL web sites.

Polices regarding email and contact

For all questions related to the course materials (lectures, assignments, and quizzes) students must use the forum of the OWL web sites (see above) instead of contacting the instructor or the TA by email. Indeed a question about the lectures or an assignment problem is likely to be of interest to many students in the class.

Writing to the instructor or the TAs should only be done for handling a personal matter, like individual projects.

Delivery mode

The format of the course lectures and tutorials will be on-line asynchronous presentations posted on the OWL web sites.

Office hours will be one-line synchronous meetings on Zoom. The office hours will not be recorded.

The slides presented during the lectures will be posted on the course web site.

Protection of online content

The instructor owns the intellectual property (IP) of the lecture materials even when such materials are posted online. Students are not to post lecture materials including assignment statements, assignment solutions, etc. to any other websites or platforms or use the lecture recording or materials for any other purpose.

Participants in this course are not permitted to record the Zoom sessions, except where recording is an approved accommodation, or the participant has the prior written permission of the instructor.

Computer requirements

Students must have a reliable internet connection and computer that are compatible with online learning (including working microphone and webcam) and testing system requirements.

As for web browser, using Mozilla Firefox v20.0 or Higher is required.

Lecture Topics

The list of topics will be something on the order of:

1. Overview of parallel and distributed computing
2. Programming patterns in parallel and distributed computing; illustration with Julia
3. Hierarchical memories, cache complexity
4. The fork-join model and the cilk concurrency platform
5. Multi-threading parallelism and performance
6. Pipelining
7. Scheduling and Synchronizing; parallelism overheads
8. Parallel Random-Access Machines.
9. Many-core programming (GPGPUs)
10. High-Performance Computing with CUDA
11. Multiprocessed parallelism, message passing (MPI)

Class Schedule

The materials (slides, recordings and other resources) of each lecture topic will be made available to the students in the course of Monday of each week of class, except for the reading week and the last week of classes.

Those materials will be accessible on the OWL web sites. Slides will also be posted on the public web site of the course.

Each student is expected to watch all the lecture recordings. Reading the slides may not be sufficient to fully comprehend the materials.

Student Evaluation

• Evaluation methods are quizzes, assignments and projects.
• Assignments and projects constitute 1/3 and 1/3 of the course mark, respectively. There is no midterm examination and no final examination either. However, there will be at least three quizzes. Quizzes constitute the last 1/3 of the course mark.
• Assignments consist of theoretical and programming exercises.
• CS 4402 projects can be theoretical work (research article presentations), practical (programming, experimentation and analysis) or a combination of both. CS 9635 projects are more advanced and must deal with current research topics, see details below.
• A quiz is a series of short and simple exercises similar to the examples of the course notes, whereas assignments consist of more advanced exercises.
• In order to successfully complete the course, a student must achieve at least 50% on assignments, 50% on projects and 50% on quizzes. This is to prevent a serious lack of effort in either area. Thus, a successful student cannot get 100% on assignments and projects and pass on the quizzes.
• A CS4402 project topic is chosen by the student from a list of topics proposed by the instructor. Project topics will be posted by February 29 and each student must choose a project topic by March 14 (at the very latest). The projects will be presented on-line synchronously by the students during the last two weeks of class, that is, the weeks of March 24 and March 31. Each project presentation runs for 15 minutes including 3 minutes for questions. Because of the class size, it is expected, but not confirmed yet, that the total time duration of the presentations will be around 8 hours, which would be divided in a number of sessions, most likely 2 hours each.
• For CS9635, there is no Assignment Two, while the Project is of much larger scale and starts during the CS4402 Assignment Two period. CS9635 projects must deal with current research topics (in the scope of the course contents) and can be related to the student thesis (in fact, this is recommended). The topic of each CS9635 project must be discussed individually with the instructor during the Assignment Two period. This implies a literature review to be done by the student and presented to the instructor during a face-to-face meeting. By March 14 (preferably earlier) the objectives of each CS9635 project must be well-defined by the student and approved by the instructor. Together with the literature review, this will give the mark for Assignment Two. Projects must then be implemented and will be presented to the CS9535 class during the Final examination period, that is, between April 11 and April 30.
• With this definition of the CS9635 project and CS9635 Assignment Two, the mark allotment is the same as for CS4402. That is, assignments, projects and quizzes constitute 1/3, 1/3 and 1/3 of the CS9635 mark, respectively.

Assignment/Project/Quiz Schedule

All dates are tentative and currently subject to change, although it is doubtful by any significant amount.

If for any reason the schedule given above cannot be adhered to, the assignment, project and quiz marks will be pro-rated. For instance, if an assignment has to be cancelled for any reason, the remaining assignment weight will be prorated to add up to 1/3.

Every effort will be made to have assignments, projects and quizzes marked and handed back within 3 weeks of the hand-in date, preferably sooner.

Assignments

Assignments will be due on the (tentative) dates listed above.

Extensions will be granted only by the course instructor. If you have serious medical or compassionate grounds for an extension, you should take supporting documentation to the office of the Dean of your faculty, who will contact the instructor.

The submission of assignments and quizzes is electronic using OWL.

Any changes, updates, and clarifications to assignments will also be posted on the course and OWL websites. It is your responsibility to monitor these pages closely.

When a student submits an assignment for evaluation, this student automatically certifies that the material she/he has handed in is exclusively her/his own work.

It is your responsibility to keep up-to-date backups of assignment disk files in case of system crashes or inadvertently erased files. Retain disk copies of all material handed in, as well as the actual graded assignment, to guard against the possibility of lost assignments or errors in recording marks. It is not safe to discard these materials until you are satisfied that your final mark for the course has been computed properly.

Academic accommodation for medical illness

If you are unable to meet a course requirement (assignment due date) due to illness or other serious circumstances, you must provide valid medical or supporting documentation to the Academic Counselling Office of your home faculty as soon as possible.

For policy on Academic Consideration for Student Absences, follow this link, and for the Student Medical Certificate (SMC), follow this other link.

There are no make-up quizzes. If your faculty's Academic Counselling Office has approved your circumstances, the value of the missed test or quiz will be reallocated.

Accommodation policies

Students with disabilities work with Accessible Education (formerly SSD) which provides recommendations for accommodation based on medical documentation or psychological and cognitive testing. The Academic Accommodation for Students with Disabilities policy can be found here.

Religious accommodation

Students should consult the University's list of recognized religious holidays, and should give reasonable notice in writing, prior to the holiday, to the Instructor and an Academic Counsellor if their course requirements will be affected by a religious observance. Additional information is given in the Western Multicultural Calendar.

Academic policies

The website for Registrarial Services is here.

In accordance with this policy. the centrally administered e-mail account provided to students will be considered the individual’s official university e-mail address. It is the responsibility of the account holder to ensure that e-mail received from the University at his/her official university address is attended to in a timely manner.

All of the remote learning sessions for this course may be recorded. The data captured during these recordings may include your image, voice recordings, chat logs and personal identifiers (name displayed on the screen). The recordings will be used for educational purposes related to this course, including evaluations. The recordings may be disclosed to other individuals participating in the course for their private or group study purposes. Please contact the instructor if you have any concerns related to session recordings.

As mentioned above, participants in this course are not permitted to record the sessions, except where recording is an approved accommodation, or the participant has the prior written permission of the instructor.

Ethical conduct

Scholastic offences are taken seriously and students are directed to read the appropriate policy, specifically, the definition of what constitutes a Scholastic Offence, at this website.

Computer-marked, multiple-choice tests and exams may be subject to submission for similarity review by software that will check for unusual coincidences in answer patterns that may indicate cheating. In particular, checking software, such as MOSS , Turnitin or others will be used randomly for detecting similarity in assignment submission.

All assignments are individual assignments. You may discuss approaches to problems among yourselves. However, the actual details of the work (assignment coding, answers to concept questions, etc.) must be your individual effort. Assignments that are judged to be the result of academic dishonesty will, for the student's first offence. You are responsible for reading and respecting the Computer Science Department's policy on Rules of Ethical Conduct and Scholastic Offenses.

Support services

Please visit the Science & Basic Medical Sciences Academic Counselling we bpage for information on add/drop courses, academic considerations for absences, appeals, exam conflicts, and many other academic related matters.

Western University is committed to a thriving campus as we deliver our courses in the mixed model of both virtual and face-to-face formats. We encourage you to check out the Digital Student Experience website to manage your academics and well-being. .

Learning-skills counsellors at the Student Development Centre. are ready to help you improve your learning skills. They offer presentations on strategies for improving time management, multiple-choice exam preparation/writing, textbook reading, and more. Individual support is offered throughout the Fall/Winter terms in the drop-in Learning Help Centre, and year-round through individual counselling. See also the services provided by the University Students’ Council.