• About
• Publications
• Courses
  • CMSC 455/655
  • CMSC 483/691
  • CMSC 313
  • CMSC 203
• Parallel I/O
  • threaded copy
  • Freeloader
  • Concurrent cURL
• ARRIA
  • Parallel Patterns
  • Task graphs
• Distributed Algorithms
  • Graph Algorithms
  • Bioinspired Algorithms
  • Distributed Neural Net.
• Program Listing

Supercomputer(n): “A computer which turns a CPU-bound problem into an I/O-bound problem” -- Ken Batcher

Here is a listing of some of my efforts in writing parallel I/O tools.
The basics involve splitting a file into pieces then distributing and receiving the pieces in parallel then reassembling the file, often from multiple computers and/or disks.
This process will always be the slowest piece of any supercomputing system, which is why it’s performance and levels of parallelism is vital.

threaded copy: A tool that uses a parallel memory mapped I/O technique to move large files on multicore systems with hundreds of GB memory.

Freeloader: A high performance storage scavenging infrastructure for networked workstations and scientific data, demonstrated fast, scalable parallel data striping.

Concurrent cURL: Using cURL, this script uses multiple nodes of a cluster to get pieces of a single file through ftp. This is the easiest way to saturate your available download bandwidth.