MCS572 UIC Cray User's Local Guide to
NPACI Cray T90 Vector Multiprocessor
and T3E Massively Parallel Processor

• T90 Overview.
  • T90 Processing Units.
  • T90 Pipelines.
  • T90 Benchmark Performance.
  • T90 Memory Units.
  • T90 UNICOS Operating System.
  • T90 Login and File Transfer.
  • T90 Programming Languages and Compilers.
  • T90 Editors.
  • More T90 Information.
• T3E Overview.
  • T3E Processing Units.
  • T3E Benchmark Performance.
  • T3E Memory Units.
  • T3E Operating System.
  • T3E Access Via T90 Front-End.
  • T3E Programming Languages.
  • T3E Message Passing Interface.
  • T3E Batch Queuing .
  • More T3E Information.
• Supercomputer Centers Overview.
• Guide Notation.

• ftp File Transfers at the NPACI Crays.
• ftp File Transfers from UIC UNIX .
• ftp File Transfers from the UIC PC Labs.
• ftp File Transfers at UIC.

• Example 1: Execution using the Terminal for Input and Output.
• Example 2: Execution using Input and Output with Files.
• Modifications for C: Compile and Execution with C.

• UNIX Log In and Out Commands.
• UNIX Information Commands.
• UNIX C Language Commands.
• UNIX makefile Commands.
• UNIX Directory Commands.
• UNIX File Commands.
• UNIX Mail Commands.
• UNIX Control-Key Commands.
• UNIX Terminal Environment Commands.
• UNIX Process Commands.
• UNIX Editor Commands.
  • The ex Editor.
  • The vi Editor.

• UNICOS Special Information Commands>
• T90 UNICOS f90 Compile, Load and Execution Commands.
• UNICOS C Language Commands.
• UNICOS Performance Commands.
• UNICOS makefile Commands.
• UNICOS Mail Commands.
• UNICOS Network Queueing System (NQS).

• UNIX Interrupts Dictionary.

• T90 Fortran90 (f90) Compiler Options.
• T90 Fortran90 (f90) Miscellaneous Extensions.
• Fortran90 Array Construction Functions.
• Fortran90 Array Reduction Functions.
• Fortran90 Array Manipulation Functions.
• Fortran90 Array Location Functions.
• Fortran90 Array Matrix Multiplication Functions.
• Fortran90 Array Functions TEST CODE.
• T90 Fortran90 (f90) Library Functions.
• T90 Fortran90 (f90) Compiler Scalar Optimization Directives.
• T90 Fortran90 (f90) Compiler Loop Directives.
• T90 Fortran90 (f90) Compiler Storage Directives.
• T90 Fortran90 (f90) Compiler Diagnostic Directives.
• T90 Fortran90 (f90) Multi-Tasking Options.

• T90 Fortran90 (f90) Timing Utility Functions.
• cc Timing Utility Function.
• Table of T90/T3E Timers.
• T3E MPI Wall Timer.

This User's Local Guide is intended to be a sufficient, hands-on introduction to the San Diego Supercomputing Center/National Partnership for Advanced Computational Infrastructure (SDSC/NPACI) Cray T90 (Cray Y-MP T916/14512) Parallel Vector Processor (PVP) and Cray T3E (Cray T3E-600 LC256) Massively Parallel Processor (MPP) for our MCS 572 Introduction to Supercomputing class. The T90 and T3E are tightly coupled at NPACI, forming a heterogeneous computing environment. Both the T90 and T3E have variations of a Unix operating system. Cray, Inc., the current name of the original company Cray Research, Inc., was reformed when it was bought out from Silicon Graphics, Inc. (SGI) by Tera Computer Company with the merged company taking the on the famous Cray supercomputing name, now the company is called Cray, Inc.. Tera developed the Multi-Threaded Architecture computer and its first machine is the MTA supercomputer node at SDSC/NPACI. The NPACI Class Account for MCS572 Fall 2000 is `UIL203'.

The T90 is a pipelined vector multiprocessor type of supercomputer has 14 vector processors. The T90 is also called a Parallel Vector Processor (PVP) by Cray and NPACI. The T90 is also called an SMP (Shared Memory Processor (SMP) or more loosely Symmetric Multiprocessor). See the NPACI official users guide: Using the NPACI Cray T90. Also see the NPACI slide tutorial: Cray T90/J90 Hardware Overview. What does the CRAY T90 look like? CRAY T90 Picture.

The T90 multiprocessor uses Cray Research Incorporated custom silicon CPUs with a clock speed of 440 MHz or clock period of 2.273 nanoseconds, and each processor has a peak performance of 1.7GigaFlops. Each has 8 vector registers with 128 words (vector elements) of eight bytes (64 bits) each. The memory on this traditional Cray is all physical memory, i.e., Seymour Cray designed machines with ``real memory'', no virtual memory or associated memory paging. Cray T90 single precision uses IEEE single precision floating point format, a four byte or 32 bit word, but only 23 bits are used for the fraction in any word (by contrast, the older Cray PVPs like the C90 used 64 bits for the `real' data type called Cray Floating Point (CFP) Format, closer to IEEE double precision; the fraction occupies 3 bytes of an IBM single precision word of 4 bytes). Cray T90 IEEE floating point double precision is 64 bits with a 52 bit fraction.

There are 2 vector fetch pipes and one vector store per processor. The processor has a vector length (VL) scalar register to hold the current vector length in the vector registers and a 128-element vector mask register for conditional storage calculations. Each CPu has chainable dual vector pipes to handling odd and even numbered operands in alternate pipes for vector addition and scalar-vector multiplication, leading to an asymptotic speed up of 4 due to the dual pipes. The independent scalar, vector and floating point function units (e.g., add, multiply with division and square root performed in hardware (Note: there is no divide pipeline on the classic Crays like the C90, so division used both reciprocal and multiply floating point function units, so the Cray C90 will give slightly different results than divide pipelined computers)) are pipelined, delivering one result per clock cycle after start-up, and they can be chained and overlapped. For instance the start up or the number of stages of the vector add pipe is 6, leading to an asymptotic speedup of six.

The T90 Cray Y-MP T916/14512, installed in 1997, ranks as the 328th top computer in the world and

The T90 has Hockney Linear Model (see MCS572 class notes) parameters of Rmax=17.29GF, Rpeak=24.78GF, Nmax=unlisted and N1/2=unlisted, for linear algebra, according the Top500 Report of June 1998, however the T90 no longer appears in the June 2000 report (see the Top 500 Class Summary). Similarly, Gunter's World's Most Powerful Computing Sites of August 1998 list the T90 at 28 GF, but does not appear in the August 2000 report (see the Most Powerful Computing Sites Class Summary). NPACI lists a peak performance of 1.76 GF per processor which would imply 24.6 GF with ideal parallelization using all 14 processors.

TOP500 Supercomputer Sites at Netlib (June 2000). {Caution: a super large data set!}

World's Most Powerful Computing Sites.

Caution: Theoretical peak values will depend on who is reporting them, since there are several different models used.

The shared main memory (RAM, called common memory (CM) by Cray) of the T90 is 512 Megawords (MW) or 4 Gigabytes (8 bytes per word, called double words on most non-Cray systems) with a peak memory bandwidth of 450GB/sec. Each word being 64 bits, arranged in a total of perhaps 256 banks with a bank cycle time of 7 clock periods, and each processor has 5 parallel memory ports (two fetch (2 words wide), one store (2 words wide), one for the 8KB-8Way scalar cache (1 word wide) and one instruction (1 word wide, for code)), with a 21GB per sec per CPU or 294GB/sec per T90 theoretical maximum memory bandwidth, but only 196GB per sec total achievable. The T90 YMP uses B and T registers for temporary stores to get around the lack of a cache. The Cray YMP has gather-scatter in hardware for indirectly addressed vectors. However, the default home directory hard disk memory size available to a user is less than 60 MB at NPACI, while the temporary work directory `/work/[npaci-login-id]' is limited to a huge 450 GB and subject to purges in 96 hours without backups. The T90 also has long term storage in the HPSS high performance solid state storage system that is used for storing super size files with an FTP-like command `pftp'. For more information on the NPACI T90 try the command "target" on "t90.npaci.edu" ( Click for Annotated target output).

UNICOS is the UNIX Cray Operating System and is a variant of UNIX with Cray extensions. Current Cray parameters can be found with the UNICOS target command with explanations of the output units using the man target command. This local guide assume the user is using the user oriented Unix C-Shell (/bin/csh), otherwise the user will have a lot of problems trying to use other shells like `sh' (Bourne) or `ksh' (Korn Shell).

Shell Caution:  If you have trouble with the above compiler command syntax, it may be that you have your borg account shell set as the Bourne shell (ksh) rather than the user standard C-shell (csh). You can change you shell for the current session by the command

/bin/csh

or you can change the shell permanently to a C-shell for which this guide is written by the command:

chsh   [user_name]   /bin/csh

where `[user_name]' must be replaced by your actual borg account name, but you must also logout and logback in again to activate the shell change. Using a C-shell should make your access must easier, while the Korn shell (ksh) or generic shell (sh) would be more powerful if you were doing computer systems work rather than scientific computing. At ACCC, new student accounts are set up with a Korn shell (ksh) in the ACCC template, probably because that is what the systems people use.

ACCC Page for UIC HP9000 (borg) features.

T90 Secure Login:  Access to the T90, called `t90' MUST be by the Secure Shell SSH Protocol, which can readily be found on the UIC student computer server `icarus' by typing:

ssh   t90.npaci.edu -l [npaci_user_id]

ssh   [npaci_user_id]@t90.npaci.edu

command, where for MCS572 your `[npaci_user_id]' is of the form `ux4526??' for Fall 2000 with `??' is a unique pair of digits. SSH works like the Unix remote login command `rlogin', but encrypts your password so that it is nearly impossible to steal. The commands `rlogin' or `telnet' do not work with the `t90', resulting in the response ``t90.npaci.edu: Connection refused''. Secure Shell ssh sets up a key on your accessing computer for t90.npaci.edu, so the first time you may be asked to `continue' to establish this key and should respond with `yes' (a simple `y' will not work). The key should work fine thereafter, unless the hostname file is corrupted or the key expires. For more ssh information see

SDSC/NPACI SSH Security Page.

Secure Access Documentation.

T90 File Transfer:  File transfer to `t90', while prohibited when accessing `TO' the `t90', is accomplished by the universal TCP/IP Internet Protocol command ONLY FROM the `t90':

ftp   [my_uic_computer].[my_department_id].uic.edu

to your UIC computer. It may be possible to use `sftp' from your UIC computer, but that is presently too difficult.

If available, secure copy `scp' (related to the Unix remote copy `rcp' command, except `scp' protects passwords) will work for SCP to or from the T90:

{To SCP from the NPACI T90 to UIC, Enter:}

``t90% '' scp [t90-filename] [uic-id]@[node].uic.edu:[uic-filename] (CR)

{To SCP from UIC to the NPACI T90, Enter:}

``t90% '' scp [uic-filename] ux4526[??]@t90.npaci.edu:[t90-filename] (CR)

{You may also place directory name in front of the target filename, e.g., `[uic-id]@[node].uic.edu:[directory]/[uic-filename]'.}

The usual programming languages for the T90 are Cray fortran Fortran 90 (f90), Cray Standard C (cc), and Cray C++ (CC), with the degree of vectorization being about the same for cc and f90.

T90 users have large storage quotas (less than 30 MB) in their T90 home directories. For much larger files, there are the users' work directories `/work/[npaci-user-id]' where `[npaci-user-id]' for MCS572 is of the form `ux451401' for Prof. Hanson, as an example. Here is a typical Fortran compile command:

f90   -O3 -r3 -o [executable] [program].f &

which uses level 3 optimization `-O3' (expands to `-h inline3, scalar3, task3, vector3' options), creates a listing file `[program].lst' from the enabled report level 3 option `-r3' passed to the compiler, and executable named `[executable]' in place of the default `a.out' from the source file `[program].f' in the background (`&'). The default optimization level is `-O2' which is moderate, level 1 scalar optimization, no parallel optimization and level 2 vector optimization. However, `-O0' forces no optimization, while for levels 2 and 3 the scalar, inlining and vector optimizations must be separately specified with then own `-O' option. Optimization level `O3' is recommended for most programs. Somewhat similarly, the usual C (or C++ with CC) procedure is

cc -O3 -h scalar2 -h report=istvf -o [executable-file] [program].c &

which creates a compiler information listing file `[program].V' (`f' suboption of `-h report' with compiler information on inlining (`i'), scalar (`s'), tasking (`t': parallel), and vector (`v') optimization, but it only gives optimization message, unlike f90 which gives other compiler information listing directly on the source), and executable `[executable-file]' in place of the default `a.out' from the source file `[program].c' in the background (`&') using higher level optimization `O3' and the `h scalar2' avoids an memory abort problem caused by `scalar3' scalar optimization. Note that moderate optimization is slightly different than in the Fortran case since the `-h' option can control optimization suboptions separately, similarly for optimization levels `-O0', `-O1' and `-O3'. The default optimization level is basically level 1 scalar optimization.

The usual editors on the T90 are the Unix visual editor `vi' and the line editor `ex'. Also, X-Windows pass through emulations are available. For extensive program revision, it may be advisable to ftp your program file back to your UIC or home computer for editing and returning it back to the T90 by ftp when finished.

For more information consult the Using the NPACI Cray T90.

The Cray T3E MC512 is a massively parallel processor (MPP) with 272 CPU processors, called PEs or Processing Elements, connected according to a three-dimensional torus topology (originally the T3D model, but with processor upgrade became T3E with the `E' only meaning that it followed the letter `D' or `enhanced'). The T3E is also called a MIMD (Multiple Instruction, Multiple Data) computer by Cray. The NPACI T3E's internet name is

t3e.npaci.edu

with the prompt nickname of `golden %'. For T3E information from NPACI, see

• Using the NPACI CRAY T3Es

Each CPU or processing node or processing element (PE) is a Digital Equipment Corporation DEC Alpha 21164 RISC 64-bit microprocessor with the nodes running at clock speed of 300 MHz (3.33ns. clock period). Each PE is a 64bit processor, it operates on words 8 bytes wide, and all 32 bit operations are performed as if 64 bit. The 272 nodes are classified into three types including 5 Operating Systems PEs, 7 Command PEs and 260 user Application PEs, with varying size memory (RAM) measured in Mega Double Words (MW). Each processor has a peak performance of 600 MFlops. Only 260 of the 272 PEs are reserved for running applications, since 5 are assigned to the global operating system and 7 are reserved for interactive sessions or single PE command applications.

The Cray T3E-600 LC256-128, installed at NPACI in 1996, ranks as the 116th top computer in the world (June 200; was the 36th in June 1998) and has peak aggregate speed of 153.6GF on linear algebra benchmarks with Hockney Linear Model (see MCS572 class notes) parameters of Rmax = 117 GF, Rpeak=268GF according to the June 2000 Top500 reports (Nmax=59,904 and N1/2=8,832 from June 1998 reports) given at the web link above. .

The memory is a hybrid logically-shared and physically distributed memory. The NPACI has a T3E-600 Model LC256-128 with 260 application processing elements, each having 16 MegaWords (128MB) each of 64bit double words making a total of 32.5GB (1998) with the peak memory bandwidth of 720MB/s and memory cycle time of 70 ns. The T3E has data, instruction and secondary memory caches.

The operating system kernel is call CHORUS, a small set of UNIX-like multiprocessing primitives. However, since access to the T3E is by remote scheduling from the T90 using the UNICOS micro kernel operating system (version UNICOS/mk 2.0.0) and the Network Queueing System (NQS), the user should refer to subsections on those topics.

Users MUST access the T3E directly using the Secure Shell (ssh), such as from UIC `icarus',

ssh t3e.npaci.edu -l [npaci-login-name]

ssh [npaci-login-name]@t3e.npaci.edu

command. SSH works like the Unix remote login command `rlogin', but encrypts your password so that it is nearly impossible to steal. The commands `rlogin' or `telnet' do not work with the `t90', resulting in the response ``t90.npaci.edu: Connection refused''. See section on the T90 for more information and how to get ssh on your own computer.

However, the user's home directory ``/usr/users/[n]/[username]'' is limited to source files and scripts totaling no more than 10MB. For compiling or linking or executing interactive jobs, the user must copy the source to the user's work directory in ``/work/[username]''. The ``/work'' directories have a ``File Wiper'' lifetime of about 80 hours, but files can be quite large, with a 171GB total for everyone. There is also the HPSS high performance solid state storage with requires ftp like commands.

The T3E programs are compiled directly on the T3D using MPP versions of the Fortran90 compiler:

f90 -O3 -r3 -Xm -o [executable] [source].f &

or the C compiler

cc -O3 -h scalar2 -h report=isvf -Xm -o [executable] [source].c &

In the above compilation commands, ``-O3'' is a high level of optimization (in the f90 case see T90 Fortran Optimization), the optimization information reports are requested through the options ``-r3'' and ``-h report=isvf'' yield ``[source].lst'' compiler informational listing for f90 and ``[source].V'' for f90 and cc compilers, respectively. Note that the and task (t) suboptions have no effect on the T3E cc compiler as in the T90 cc compiler. The option ``-Xm'' is ESSENTIAL so that a ``malleable executable'' is produced that can be executed with any number of processors with ``mpprun'', while the form ``-X[npes]'' will cause problems for ``mpprun''. The library option ``-l mpi'' is already assumed as a default by the compilers and permits use of MPI parallel programming in the code.

Execution of the executable ``[executable]'' is by the massively parallel envelope command:

mpprun -n[npes] [executable] < [data] >& [output] &

where the user is restricted to one non-production, interactive job at a time using no more than 32 PEs for not more than 60 minutes, while ``[data]'' and ``[output]'' are optional input data and output files, respectively.

The massively parallel envelope command is `mpprun', which assumed that the executable is malleable, i.e., the `-X [npes]' processor number option has not been used with the compile commands `f90' or `cc'. The compilers and mpprun do not require any additional libraries to handle MPI commands.

The T3E can be used interactively for smaller jobs at most times, or it can be used through the NQE (NQS) batch system, with the new version of parallel programming language called MPI (Message Passing Interface) using the latest and greatest MPT (Message Passing Toolkit) with new default buffer size.

There is also an another nonCray MPI execution envelope command called ``mpirun'' that has a similar format"

mpirun -np[npes] [executable] < [data] >& [output] &

but note that the number of processors option is ``-np'' rather than ``-n''. It is not clear what the advantage is over the Cray ``mppun'' command, except that it allows an interface with the ``p4'' macros which are related to shared memory ancestors to ``mpi''.

Simple job status can be obtained from the jobs command which will display ``Running'' or ``Done'' for normal operation, or ``Exit'' for error aborted jobs; the generic process status ps command will list all processors with cpu timings that you have running on your job. The advanced process status for all used application processors can be found with the ps plus command:

The parallel programming parts of these MPP language versions are with processor communication handled by a new version (5/2000) called the Cray Research MPT (Message Passing Toolkit) with components

MPI (Message Passing Interface)

or

PVM (Parallel Virtual Machine)

or

SHMEM (Logically Shared, Distributed Memory (SHMEM) Routines)

forms of message passing embedded in the Fortran or C code. Sample T3E interactive MPI programs for Jacobi iterations of Laplace's equation are given in for the revised Fortran 90 code:

• laplace.mpi_f.f

• laplace.mpi_f.output for 200X200 mesh and 3000 maximum iteration.

and a one for the revised C code is given in

• laplace.mpi_c.c

• laplace.mpi_c.output for 200X200 mesh and 3000 maximum iteration.

For more MPI information, see the MCS572 MPI Class Pages:

• MCS572 NPACI Cray MPI General Information Page.

• MCS572 NPACI Cray MPI Example Page.

• Getting Started With MPI: A Message Passing Interface for Parallel Programming: An Introduction to MPI at SDSC.

• BU MPI HTML Man Pages.

Remote job scheduling on the T3E is accomplished by using the NQS (Network Queueing Environment or System) job scripts. A sample PSC T3E target job script for Fortran code is given in

fpgm.job

and a one for PSC T3E C code is given in

cpgm.job

These job scripts are run with the NQS QSUB submit command from the user's `${TMP}' temporary directory, such as by

qsub ${HOME}/fpgm.job    or    qsub ${HOME}/cpgm.job

where `${HOME}' denotes the meta-name of the user's home directory on `t90'. The job status can be checked by the NQS QSTAT status command:

qstat -a

and when done, the user can view the output if any. If for any reason you need to kill the job before the end, first note the job id number `[job_id]' at the beginning of your job line in the `qstat -a' output, then enter the command:

qdel -k [job_id]

A user can try out the class sample NQS QSUB job scripts by copying one of the above Laplace code to your home directory and then recopying it to the recyclable source file of the form `*pgm.*' as follows:

cp laplace_mpi_f.f fpgm.f    or    cp laplace.mpi_c.c cpgm.c

depending on whether you want to test Fortran or C versions; you will also have to create a simple input data file called `data', inserting the number of iterations (e.g., using vi to insert: 200) into the input data file; then in the home directory entering the queue submit command:

qsub fpgm.job    or    qsub cpgm.job

then check for a finished job with `qstat -a' until the message "no batch queue entries" appears, finally looking for output in a file of the form `*pgm.output'. You can always modify the sample job scripts to suit your particular job requirements.

There is also the newly installed Portland Group High Performance Fortran (command = pghpf), so use `man - pghpf' to find out how to use it (untested by FBH).

For T3E information from NPACI, see

• Using the NPACI CRAY T3Es

Pittsburgh and Cornell Center have been phased out of the NSF centers. Users will have to switch as early as March 1998 to NCSA in Urbana which has a Cray Origin 2000 or SDSC in San Diego which has both Cray T90 and T3E. MCS572 Fall 1998 had access to both NCSA-NCSA and NPACI-SDSC. MCS572 Fall 2000 with use the NPACI-SDSC Cray T90 and T3E.

This mini-local-guide is meant to indicate ``what works'' primary for access from UNIX systems to NPACI-SDSC. The use of the Unix C-Shell on the T90 is assumed throughout most of this local guide.

Cray, CF77, CFT77, Auto-tasking, SEGLDR, and UNICOS are trademarks of Cray Research, Inc. UNIX is a trademark of AT&T.

Computer prompts or broadcasts will be enclosed in double quotes (``,''), background comments will be enclosed in curly braces ({,}), commands cited in the comments are highlighted by single quotes (`,') {do not type the quotes when typing the commands}, and optional or user specified arguments are enclosed in square brackets ([,]) {However, do not enter the square brackets.}. The symbol (CR) will denote an immediate carriage return or enter. {Ignore the blanks that precede it as in `[command] (CR)', making it easier to read.} The symbol (Esc) will denote an immediate pressing of the Escape-key {Use no brackets please.} The symbol (SPACE) will denote an immediate pressing of the Space-bar {Warning: Do not type any of these notational symbols in an actual computer session.}

For further information, please consult the sources (you can just click on the highlighted topics to access if you are surfing the world Wide Web):

1. Professor Hanson's MCS 572 Introduction to Supercomputing Home Page provides a large variety of links to useful supercomputing information.

2. San Diego Supercomputing Center (SDSC/NPACI)Home Page on the World Wide Web permits the direct search of the NPACI public web information directories. Use of Mosaic, Netscape, OmniWeb or lynx (line edit version) applications permits this access. For a faster introduction to NPACI see of HotPage Information.

3. NPACI User HotPage.

4. NPACI Documentation Page.

5. Using the NPACI CRAY T90 provides access to a great deal of information on the T90 including specifications and a good size index.

6. Introduction to Vector Computing, NPACI-SDSC Slide Tutorial.

7. Cray T90/J90 Hardware Overview, NPACI-SDSC Slide Tutorial.

8. T90 Fortran Optimization, explaining `f90' optimization levels much better than `man - f90' and many other things.

9. CRAY (SV1-T90) Course Notes
   University of Western Ontario, Information Technology Services:
   
   1. Brief Intro to make
   2. Brief Intro to RCS
   3. An Introduction to High Performance Computing
   4. How to use the Cray
   5. Cray Programming Notes
   
   5.1. Introduction to Performance Analysis Tools on Cray
   5.2. Vectorization
   5.3. Parallelization
   5.4. Memory Access
   6. Introduction to Fortran 90
   7. Introduction to Threads

10. CRAY T90 Product Information (Cray T90 Homepage).

11. Using the NPACI CRAY T3Es provides access to a great deal of formation on the T3E including specifications and a good size index.

12. CRAY T3E Series Product Information (Cray T3E Homepage).

13. CRAY T3E Technology Profile

14. CRAY Research Manuals Search (Cray CF90, C, C++, ...)

15. Cray T3E User's Guide from Finland

16. I/O Optimization on the CRAY T3E

17. Getting Started With MPI: A Message Passing Interface for Parallel Programming: An Introduction to MPI at SDSC.

18. MCS572 NPACI Cray MPI General Information Page.

19. MCS572 NPACI Cray MPI Example Page.

20. "T3E" MPP Fortran Programming Model (paper by ftp).

21. The Benchmarker's Guide to Single-processor Optimization for CRAY T3E Systems, postscript paper by Ed Anderson, Jeff Brooks, and Tom Hewitt, Cray Research, June 1997.

22. man [command] (CR), when invoked in a UNIX-like system such as UNICOS, produces an on-line listing of the manual pages on the command [command], or similar function.

23. Consultation concerning problems related to using the Crays can be obtained from Professor Hanson {718 SEO, X3-2142, hanson@uic.edu}. NPACI protocol requires that Professor Hanson contact NPACI consultants for this class, if they are necessary.

The login procedure depends on your local method of accessing the Convex from UIC, but the best access is from a Unix type system since the NPACI Cray T90 operating system is UNICOS, which is substantially Unix and it is to the user's advantage to use Unix to Unix communication. If you do not now have a Unix account you should try to get one from your department's Unix system or from the UIC Computer Center graduate student Unix (Sun) server called `icarus'. Unix workstations are available in many science and engineering departments. If that does not work out or is not practical, then see Professor Hanson about other alternatives.

Logging into the T90:

Finding Work Disk:
At any time you can determine your temporary disk path name (it is given during the MOTD):}
``t90%'' echo $WORK (CR) {This command echoes the name of your current work directory (of the form `/work/[npaci_user_name]') that is usually displayed when you log in. `echo $TMP (CR)' does the same thing. The `cd $WORK (CR)' command takes you to your temporary directory and the `cd $HOME' command changes back to your home directory. The temporary directory is limited to active files that will be there for short periods, but there is no specific quota in size. Executable and object modules can take up a lot of space, so a good temporary location for them is the work disks. Be careful not to take up valuable UNICOS disk space, erase unneeded files by the command `rm [file] (CR)'. Otherwise, continuing the session:}
``t90%'' cp   pgm.f $WORK (CR) {This command copies the a Fortran source file `pgm.f' to the user's temporary disk where the user will be less likely to run out of memory.}
``t90%'' cd $WORK (CR) {This command changes from the current directory to the user's temporary directory for doing computational intensive supercomputing.}
``t90%'' rm pgm.o run output (CR) {Removes the files named `pgm.o', `run' and `output' if they exist (answer `y' for yes if they do, or else UNICOS may say it can not find the UNICOS file name. This may be needed to prevent interference when the updates to these files are produced by compile and execute commands, while copy over restrictions are in effect.}

Movement and Alternate Ways to Execute Files:

You can instead move the files to your default home directory with the move command:}
``t90%'' mv output ${HOME}/[label].output (CR) {Moves the output file to your home directory and relabels it to `[label].output' for uniqueness, because on the temporary disk they are liable to be automatically deleted after inactivity.}

{You can execute `run' in the temporary directory while in the home directory with the command:}
``t90%'' ${TMP}/run < data > output & (CR) {Executes the executable in the temporary directory, where the mandatory curly brackets around the variable `TMP' tell the UNICOS shell that meta variable name is only `TMP' and not `TMP/run', for example. Also, while in the temporary directory, the home directory file `data' file is redirected as input to the executable `run'.}
``t90%'' run < ${HOME}/data > output & (CR) {Executes the executable from another directory other than the home directory, assuming that the input file `data' is still in your home directory, but that the output file `output' will be stored in your temporary directory in this example. The curly brackets in ``${HOME}'' are necessary, and do not signify a comment there. ~other topics are described in the Command Dictionaries of Section COMMAND DICTIONARY.}
``t90%'' logout (CR) {Logs you out of your remote NPACI session. You then return to your prior session on UICVM or UNIX or other computing seat.}

The login procedure for the NPACI Cray T3E Massively Parallel Processor is very similar to the NPACI Cray T90 Parallel Vector Processor, but the sample codes are usually different corresponding to the different types of processing models: distributed memory versus shared memory.

Logging into the T3E:

For an example of compiling and linking the MCS572 Fortran based `t3e' starter problem from your temporary directory (assuming `fpgm.f' has been transferred to your `t3e' account), enter:
``t3e% '' mkdir   /tmp/[NPACI_user_name] (CR) {This makes your temporary directory (unlike T90 you have to create your own and may have to recreate your temporary directory again is it is wiped out by the `File-Wiper. Next enter:}
``t3e% '' cp   fpgm.f /tmp/[NPACI_user_name] (CR) {This copies the starter source file to your temporary directory, (the meta-variable `${TMP}' on the T3E only denotes the temporary directory created for each batch run). Next enter:}
``t3e% '' cd   /tmp/[NPACI_user_name] (CR) {This changes the current directory (cd) to the user's temporary directory where you should compile, load and execute your code. Do not worry about your home directory since you go back by the command `cd ~'. Some users prefer the push and pop directory commands such as `pushd /tmp/[NPACI_user_name]' to get to the temporary directory and `popd' to return `${HOME}' or to which ever directory has been previously accessed, since prior directories are kept in a push/pop buffer stack. Next Enter:}
``t3e% '' f90 -O3 -r3 -Xm -l mpi -o fpgm fpgm.f & (CR) {Here, `f90' is the optimizing Cray Fortran 90 compiler, `-O3' is the level 3 optimization (usual) option, `-r3' is the enable marking compiler report option with the report going to the file `fpgm.lst', `-Xm' ensures a malleable executable that can run on any number of available T3E application processors, `-o fpgm' means that the execution file is named just `fpgm' rather than the default `a.out', `fpgm.f' is the Fortran 90 source file, and the last `&' means compilation is done in the background permitting use of the terminal while waiting for the job to finish. The status of the job can be determined form the Unix `jobs' command:}
``t3e% '' jobs (CR) {The process status `ps' command can also be used. When you get the message:
``Done   f90 -O3 -r3 -Xm -l mpi -o fpgm fpgm.f''
instead of ``+Running ...'', then the listing file can be examined by the Unix `more' paging command:}
``t3e% '' more   fpgm.lst (CR) {Else, the listing file can be viewed by the Unix visual editor (or other favorite editor):}
``t3e% '' vi   fpgm.lst (CR) {If compilation and listing is satisfactory, then create a input file called `data' with the maximum number of iterations `5000' (without quotes) in it and is needed by the executable:}
``t3e% '' vi   data (CR) {and the module `fpgm' may be executed within the `mpprun' envelope run environment on 4 T3E processors:}
``t3e% '' mpprun -n4 fpgm < data >& fpgm.output & (CR) {The mpprun command supplies a copy of the executable to each of the 4 processors (`-n4' option) needed for this code and all 4 participate in the solution, synchronized by the MPI parallel programming library subroutines. Again, the job status may be checked while waiting (or using `ps' which should indicate how long 4 processors have been running or `psp -p app' for much more information) by entering:}
``t3e% '' jobs (CR) {When the ``Done'' message is displayed rather than ``+Running ...'', then the output can be viewed in pages:}
``t3e% '' more   fpgm.output (CR) {Caution: the executable `fpgm', as with the default executable `a.out', is a binary, rather than text file, so is not readable. Next you can transfer your files back to your printer connected computer (NPACI would be prefer that you not print from `t3e' since you do not have a way to retrieve the output) by FTP:}
``t3e% '' ftp   [home_machine].[department].uic.edu (CR)
``login: '' [user_name] (CR)
``password: '' [user_password] (CR)
``ftp> '' cd   [target_directory] (CR)
``ftp> '' put   fpgm.f (CR)
``ftp> '' put   fpgm.lst (CR)
``ftp> '' put   fpgm.output (CR)
``ftp> '' by (CR)

If available, secure copy `scp' (related to the Unix remote copy `rcp' command, except `scp' protects passwords) will work for SCP to or from the T3E:

{To SCP from the NPACI T3E to UIC, Enter:}

``t3e% '' scp [t3e-filename] [uic-id]@[node].uic.edu:[uic-filename] (CR)

{To SCP from UIC to the NPACI T3E, Enter:}

``t3e% '' scp [uic-filename] ux4526[??]@t3e.npaci.edu:[t3e-filename] (CR)

{You may also place directory name in front of the target filename, e.g., `[uic-id]@[node].uic.edu:[directory]/[uic-filename]'.}

{Finally for the job, it is good file management, even in the temporary directory, to remove your executable file since it is cheaper to regenerate than to store, so as not to be a "storage hog" using the Unix remove `rm' files command:}
``t3e% '' rm   fpgm (CR) {At this point you can `logout' of `t3e' by entering:}
``t3e% '' logout (CR)
``% '' {Return to your local UIC UNIX session.}
 

Processing C Code:

``t3e%'' mpprun -n4 cpgm < data >& cpgm.output & (CR) {The `mppun' command form finally executes the program `cpgm.c' on 4 processors (`-n4' option) with each processor getting its own copy of the executable, synchronized by the MPI parallel programming library functions. The output of `cpgm' is redirected into the output file `cpgm.output' in the background. This command line has the same syntax as it would for a Fortran job. The `>  cpgm.output' option would not be used if output at the screen is desirable, but that is rarely practical except for small jobs. The ending `&' means that the job will run in the background enabling the user to continue working in the session. Interactive jobs are limited to 15 minutes and 32 processors, while batch jobs (see about `nqs' in Subsection on NQS) have a much less limits. Once you have finished with the huge `cpgm' file, it is a good file management to either remove it with the command:}
``t3e%'' rm cpgm (CR) {Removes the files `cpgm' answer `y' for yes to remove the two files if queried. When you are finished on `t3e' then}
``t3e%'' logout (CR) {Logs you out of your remote NPACI session. You then return to your prior session on UICVM or UNIX or other computing seat.}

The FTP file transfer protocol is the fastest method of file transfer between UIC and NPACI Cray UNICOS, because it uses a fast internet communication link.

At the NPACI Crays you can transfer file between the Crays and UNIX. The `ftp' command on UNIX is very much like the `ftp' command in UNICOS.

In order to transfer a file from UNICOS and to UIC, enter the commands:

For Transfer to UIC Unix:

``t90% '' ftp [machine].[dept].uic.edu (CR) {where you must enter the actual machine-department "[machine].[dept]" name; {caution: some machines will have more than one part to the name}}

``Name ([machine].[dept].uic.edu:[NPACI-name]): '' [UIC-name] (CR)
``Password: '' [UIC-password] (CR) {If you make a mistake at this point and you get an `ftp> ' prompt then you can either enter `by' and start again or enter `user [UIC-name]' continuing:}
``ftp> '' cd [UIC-directory] (CR) {change directory to the target UIC directory.}
``ftp> '' lcd [UNICOS-directory] (CR) {locally change directory to the target NPACI UNICOS directory, but note that the meta variables `$HOME' or `$TMP' will not work here and you will need the full path name since FTP is an entirely different session! Hence it is best to initiate FTP from the UNICOS directory from or to which the transfer will be made. For the home directory the the abbreviations `~' or `~[name]' will still work even if `$HOME' does not.}
``ftp> '' ls (CR) {Lists files in the UIC directory; wild-cards can be used.}
``ftp> '' !ls (CR) {Lists files in the local UNICOS directory using the `!' escape to the local UNICOS session; wild-cards can be used.}
``ftp> '' !get [UIC-fn.ext] [[UNICOS-fn.ext]] (CR) {Gets file from the remote UIC directory and save in the local UNICOS directory with optional new name `[UNICOS-fn.ext]'; wild-cards cannot be used, but the ftp multiple get command `mget' does permit wild-cards.}
``ftp >'' put [UNICOS-fn.ext] [[UNIX-fn.ext]] (CR) {Puts file from the local UNICOS directory and save in the remote UIC directory with optional new name [UNIX-fn.ext] ; wild-cards cannot be used, but the ftp multiple get command `mput' does permit wild-cards.}
``ftp >'' by (CR) {Quit or Bye (by) when done transferring files.}
``t90%''

{To SCP from the NPACI T90 to UIC, Enter:}

``t90% '' scp [t90-filename] [uic-id]@[node].uic.edu:[uic-filename] (CR)

{To SCP from UIC to the NPACI T90, Enter:}

``t90% '' scp [uic-filename] ux4526[??]@t90.npaci.edu:[t90-filename] (CR)

{You may also place directory name in front of the target filename, e.g., `[uic-id]@[node].uic.edu:[directory]/[uic-filename]'.}

The file transfer protocol program from a UNIX session is a similar to file transfers from the NPACI UNICOS sessions, because both have UNIX or extended UNIX operating systems, as discussed in the last section.

File transfer protocol (ftp) on a PC Lab PC may not be practical for must users, due to lack of permanent storage. The nearest Xerox PostScript printer to 2249f is SEL2263, while others are SEL2265, SEL2058 and SEO327. However, if the PC is your favorite medium, then use it as in the above Cray or Unix subsections.

• Compilation:
  {Starting in Cray UNICOS with a f90 compatible FORTRAN source file named `[source].f', the typical format of the UNICOS compilation command is:}
  f90 -r3 [source].f (CR)
  {Here, `-r3' is the level 3 report option that enables the helpful information marking of loops for scalar optimization (S), vector optimization (V), parallel optimization (P), unrolling (R), unwound (W), or short vector (Vs) optimization in the compiler information listing file that is automatically stored in the file `[source].f'. No additional option is need for the automatic optimization, unless altered. The same thing can be accomplished by the preferred command}
  f90 -c -r3 [source].f (CR)
  {An object file is also produced and is stored in `[source].o'. }

  {For Cray Standard C source file named `[source].c':}
  cc -c -h report-istvf [source].c (CR) {This is the Cray Optimizing C compile command, with compiled source produced called `[source].o'. Here the option `-c' denotes compilation. Also}
  cc -hreport=isvf -o run [source].c (CR)
  {compiles and links the C program while producing messages on inlining, scalar and vector optimizations.}

• Linking/Loading Step Only:
  
  segldr -o run [source].o (CR) {This step is the link or load step using the Cray UNICOS segment loader with a typical format. Here the executable name is taken to be the generic name `run', but you can choose whatever name you like, omit the `-o' option produces the default UNIX name `a.out' for the executable. However, it is wise to stick to a single executable name, because the file size of typical executable will exceed your 250 Cray Block home directory limit and a single name makes it easier to delete or move to either your work disk. The `f90' combines the compile step of `f90 -c' and the link step of `segldr' such as in the command `f90 -r3 -o run pgm.f'.}

• Execution Step:

  {If you use UNIX redirection for input and output then the format is of the execute command is like:}
  run < [data] >& [output] & (CR) {where `[data]' is the input file and `[output]' is the output file that also receives diagnostic messages in the background. Input files for FORTRAN `read' statements and output files for Fortran write statements can also be allocated to the terminal or UNICOS files using the Fortran `open' statement to reallocate units 5 and 6 for f90 as in the example programs used below.}

As practice, you can run any source program that you have transported to UNICOS. The simple code ` convert.f':

      program convert
code: convert from debug fortran  cogs, slightly modified.                     
change:  input & output is to & from terminal, input at prompt.
caution: compile, load, and execute in UNICOS using the three commands:
command:   f90 convert.f
command:   segldr -o convert convert.o
command:   convert
      real a(999)
      write(*,*) 'input any integer less than 1000:'
      read(*,*) i
      a(i) = float(i)
      write(*,6000) a(i)
6000  format(' floating point representation: ',e13.5)
      write(*,*) 'What happens when you exceed array bound of 999?' 
      stop  
      end   

{Since this simple-minded `convert.f' program uses the terminal as undeclared input and output units, corresponding respectively to `read(*,*)' and 'write(*,[n])' statements, without specifying an the f90 `open' statement. Be sure to do this in your work directory which you can change to by using `cd $TMP'. The source `convert.f' is executed with the 2 commands:}
f90 -o run convert.f (CR)
run (CR) {Note that the program is too trivial to need any optimization options. Here `f90' is a utility combining both the compile and load commands (no convert.o file is stored). Upon execution with `run', you are asked to supply an integer like 6:}
``Input an integer less than 1000:''
6 (CR) {UNICOS responds with the output:}

 floating point representation:   0.60000E+01
 What happens when you exceed array bound of 999?
 STOP executed at line 15 in Fortran routine 'CONVERT'
 CP: 0.002s,  Wallclock: 2.311s
 HWM mem: 163926, HWM stack: 2048, Stack overflows: 0

{To rerun the same code without recompiling, merely enter `run' again:}
run (CR) {Your response should be to enter another number as above. Do not spend too much time with `convert.f, because it can only read and write.}

The second example uses data files for both input unit 5 and output unit 6, as well as the UNIX Fortran seconds timer `second()'

code:  craytest fortran  from cogs disk =ncsa ctss users guide eg#1
      program tempt 
calculation:  c(i)=exp(-0.5)*pi/i
change:  input from file 'tempt.data' and output to file 'tempt.output'
change:  particular input is the vector length "nx"
change:  second cpu(user) time 'second' added.
caution:  In Cray T90 UNICOS compile, load and execute with:
command:  f90 -r3 -o tempt tempt.f
command:  tempt 
      parameter (ndim=5120)
      real a(ndim),b(ndim),c(ndim)
caution:  cft "real" implies 48bits = 6bytes for the fraction,
continued:  unlike ibm "real" which implies 24bits = 3bytes.
continued:  Otherwise all variables not starting with (i-n) are
continued:  implicitly real, unless otherwise declared.  The fraction
continued:  for "double precision" is 96bits=12bytes, hence no "real*8"
      open(6,file='tempt.output')
      open(5,file='tempt.data',status='old')
      read(5,*) nx
      t1=second()
      t2=second()
      call init(ndim,nx,a,b)
      call calc(ndim,nx,a,b,c)
      t3=second() 
      clock=t2-t1
      time=(t3-t2-clock)
      write(6,66) nx,nx,c(nx),clock,time
66    format(1x,' nx=',i5,'; c(',i5,')=',f10.7
     & /'  clock=',f12.7,' seconds; code=',f12.7,' seconds')
      stop
      end
           subroutine init(ndim,nx,a,b)
comment:  removed cdir$ novector
      real a(ndim),b(ndim)
      pi=acos(-1.0)
      do 10 ix=1,nx
         a(ix)=pi
         b(ix)=float(ix)
10    continue
      return
      end 
           subroutine calc(ndim,nx,a,b,c)
comment:  removed cdir$ novector
      real a(ndim),b(ndim),c(ndim)  
      do 20 ix=1,nx
         c(ix)=exp(-0.5)*a(ix)/b(ix)
20    continue
      return
      end 

{CAUTION: you should not use `ex' or `vi' for extensive editing on the T90, which you can do at your local host, UICVM or UNIX; however, this short editing session will only take a short amount of time. A selected list of `ex' commands are given below. The `ex' prompt is `:'. You add lines with the `ex' subcommand `0a' to start with, to add one after line 0 in:}
``:'' 0a (CR)
500 (CR)
. (CR) {Finally save and exit `ex' with the combined `wq' = `w | q' subcommand:}
``:'' wq (CR) {If successful (you can check by `cat tempt.data (CR)' you should now have files called `tempt.data' from `ex' and `tempt.f' from `ftp' or other source, so that you can now compile, load, and execute `tempt.f' by entering the three command lines:}
``t90%'' f90 -r3 tempt.f (CR)
``t90%'' segldr -o run tempt.o (CR)
``t90%'' run (CR) {If successfully executed, UNICOS should respond something like:}

 STOP executed at line 31 in Fortran routine 'TEMPT'
 CP: 0.002s,  Wallclock: 0.114s,  0.1% of 16-CPU Machine
 HWM mem: 182085, HWM stack: 19416, Stack overflows: 0

{Your output will be in `tempt.output' and you can list is by the command:}
``t90%'' cat tempt.output (CR) {Your output should look something like:}

  nx=  500; c(  500)= 0.0038109
  clock=   0.0000041 seconds; code=   0.0000102 seconds

{If you wish to re-run the program again with a different number, the enter `ex tempt.f (CR)' again or `!ex (CR)', and enter within EX the subcommand `1c (CR)', type `1000 (CR)' to change `500' to `1000', enter `.' to end the change subcommand, enter `wq (CR)' after the ``:'' prompt, and then enter `run (CR)' and `cat tempt.output (CR)' again. When you are done with `tempt.f', remove all the files from UNICOS that you do not need, using:}
``t90%'' rm run tempt.o tempt.l (CR) {for example, but especially the big `run'.}

See also:

man f90 (CR)
or
man 1m f90 (CR) : for the manual help pages on the CF90 compile and load command; the option `1m' of `man' brings up the page for the T3E MPP.

For information on C language programs use the UNICOS commands:

man cc (CR) : for the manual help pages on the C compile and load command; also the `-q' option of man will produce quick, compact manual pages.

For more details on commands and typical formats refer to the subsections of Section UNIX Command Dictionary. When executing larger codes with `run', the user may have to work on the user's temporary disk. For even more details see the NPACI UNICOS STARTUP PACKAGE and the appropriate Cray manuals. Also use the UNICOS `man' command.

• UNIX Log In and Out Commands.
• UNIX Information Commands.
• UNIX C Language Commands.
• UNIX makefile Commands.
• UNIX Directory Commands.
• UNIX File Commands.
• UNIX Pipe and Redirection Commands.
• UNIX Mail Commands.
• UNIX Control-Key Commands.
• UNIX Terminal Environment Commands.
• UNIX Process Commands.
• UNIX Editor Commands.
  • The ex Editor.
  • The vi Editor.

help [SCCS command name] (CR) : This special SCCS command name to find about about return messages. See `docview' for more information.
docview [options] (CR) : This special UNICOS command permits the access to Cray online documentation, options `a[list]' for an alphabetical list of major document names such as primer, docview, UNICOS60update, c.std, fortran, parallel_5.0, segldr, perf, scc30, and cdbx, `c[list]' for a list by categories like introductory, languages and utilities, `f[ind] [string]' for searching, `h[elp] [topic]' for help, `v[iew] [docname] [keywords]' for viewing targeted passages, `w[rite] [docname] [keywords]>[docname].doc' for writing a targeted passage to a disk file named `[docname].doc'. Use `entire' for `[keyword]' if you want to write the entire document, but you should only do this on the ? temporary disk by first entering the command `cd temporary'. The command is menu driven with plenty of pointers for help and list of local commands.
usage (CR) : NPACI command to display average cpu usage chart (user, idle, system). Use `man usage' for more information.
quota (CR) : Query UNICOS about memory quotas, which are important to avoid over-running the 250 Cray Block disk limit and other memory limits.

f90 -r3 -[other options] [source].f [other source files] (CR) : Compiles source file `[source].f' and `[other source files]' with the Cray level 3 report compiler option `-r3' both with the default full optimization (`noaggress bl noinline recurrence norecursion scalar vector ....... nozeroinc'), producing an object file `[source].o' and compiler annotated listing file `[source].l' with vectorization information:



Marking                       Meaning 
   S          scalar loop optimization (major marker)
   V          vector optimization (major marker)
   P          Parallel optimization (major marker)
   Vs         short vector optimization
   W          unwound     (major marker) {short inner-most loops with trip 
              counts of not more than 5 are collapsed or transformed to single
              statements so that the next inner-most loop can be vectorized
              provided there are no dependencies}
   b          bottom loading     {pre-fetching is used for the next
              iteration of scalar loops, only and `-o nobl' kills it} 
   c          conditionally vectorized, {subject to run-time
              determination of recurrence vector length}
   k          kernel scheduling
   i          unconditionally vectorized with IVDEP
   r          loop unrolling     {a set of loop iterations is
              collapsed into one iteration that has been enabled by the `-e'
              enabling option with its `m' loop marking sub-option}
   D          delete loop

Use `-emx' in place of `-em' if you want a cross reference listing also. Use the `-b [binfile]' option to name the object file with a name other than the default `[source].o' name. Use `-o aggress' to turn on a more aggressive form of optimization, but be careful of the results. Use `-o inline' or `-I [inline-source]' to get inlining of subprograms to avoid their overhead. Use the compiler directives `NORECURRENCE' or `IVDEP' and `RECURRENCE' to turn off and on the optimization of loop recurrences. Use `-o recursion' to enable subprograms to be recursive. Use `-o zeroinc' if zero increments of do loops indices or constant increment variables (CIV) are used, because the default assumes there are none. Use `segldr' command to load the execution module, which then can be used to execute the program. See below and the last section for more on the options. It is much better to use makefiles for such commands.

f90 -eS [source].f (CR) : Creates a Cray Assembly Language (CAL) file or calfile named `[source].s' for the Fortran program `[source].f' that can be used with the Cray Assembler or to determine how the Cray compiler has carried out the optimization, particularly how it has used the vector registers. The option `[name].s' can be used to name the calfile with something other than the default name. No object or binary file `[source].o' is produced, and a nasty message will be given instead.
f90 -g [source].f (CR) : Compiles the f90 and generates a symbol table for the debugger, like `cdbx' (use `man cdbx'). See also `-G debug_lvl', where `-G 0' is the same as `-g'.
segldr -o [executable-file] -l [library-list] [source].o (CR) : This segment loader links and loads the object module `[source].o' from the `f90' step into the execution module named `[executable-file]' by the `-o' option. Without the `-o' option, the executable is the standard `a.out' file. The library option may not be needed because many libraries are searched by default: Pascal (libp.a), I/O (libio.a), utility (libu.a), Fortran (libf.a), C (libc.a), Math (libm.a), and Science (libsci.a). Numerical Recipes in Fortran or C of Press et al. are not directly available in UNICOS.
f90 [-options] -o [executable] [source].f (CR) : The `f90 -o [executable]' command combines both `f90' compile and `segldr' load functions in one command; e.g.,
f90 -limsl [source].F (CR) : This Fortran90 parallel form is for using the IMSL mathematical and statistical library; if more than one processor is used, then `setenv NCPUS [nn]' must be executed first with `[nn]' is number of CPU's requested. For more information, click on:
IMSL Software at NPACI

To find out what other special software is at NPACI click on: NPACI Installed Software


[executable-file] < [input-file] > [output-file] & (CR) : Executes the executable module taking input from the file `[input-file]' and redirecting output to `[output-file]' as a background process.

cc -o run [file].c (CR) : Compiles source [file].c, using the standard C compiler `scc2.0' and producing an executable named run. In place of `cc', use `scc3.0' or `scc' for the latest version of standard C or `pcc' for portable C.
cc -c [file].c (CR) : Compiles source [file].c, using the standard C compiler `scc2.0' and producing an object file named [file].o.
cc -hnoopt -o run [file].c (CR) : Compiles source [file].c, using the standard C compiler `scc3.0' and producing an executable file named run without scalar optimization or vector optimization while `hopt' enables scalar and vector optimization, Some other optimization related options are `-hinline' for inlining while `-hnone' is the default no inlining, `-hnovector' for no vector (vector is the default), and `-h listing' for a pseudo-assembler (CAL) listing. Some standard C options are `-htask3' for automatic parallelization (autotasking in "crayese") and `-hvector3' for more powerful vector restructuring. Other `-h' suboptions are `ivdep' for ignore vector dependence, `-hreport=isvf' generates messages about inlining (i), scalar optimization (s) and vector optimization (v), and `-hreport=isvf' writes same messages to `[file].v'. A commonly used form will be
cc -o run -h report=isvf [file].c (CR)

See `man cc' or `docview' for more information.
#define fortran : Form of C header statement to permit the call to a fortran subroutine from a C program. For example:

#include <stdio.h>
#include <fortran.h>
#define fortran
main()
{
         fortran void SUB();
         float x = 3.14, y;
         SUB(&x, &y);
         printf("SUB answer: y = %f for x = %f\n", x, y);
}


#pragma _CRI [directive] : Form of C compiler directive placed within the C code, where some example directives are `ivdep' for ignoring vector dependence, `novector' for turning off the default vectorization, `vector' for turning it back on, `inline' for procedure inline optimization, `shortloop', `noreduction', `getcpus [p]', `relcpus', `parallel ........', and `end parallel'. See `vector directives' for instance in `docview' for more information and examples.
segldr -o [executable-file] -l [library list] [source].o (CR) : This segment loader links and loads the object module `[source].o' from the `f90' pure compile step into the execution module named `[executable-file]' by the `-o' option. Without the `-o' option, the executable is the standard `a.out' file. The library option may not be needed because many libraries are searched by default: Pascal (libp.a), I/O (libio.a), utility (libu.a), Fortran (libf.a), C (libc.a), Math (libm.a), and Science (libsci.a). Numerical Recipes in Fortran or C of Press et al. are not directly available in UNICOS.
[executable-file] < [input-file] > [output-file] & (CR) : Executes the executable module taking input from the file `[input-file]' and redirecting output to `[output-file]' as a background process.

Cray Prof Profiling Facility:


Cray Error Explaining Command:

explain [error-message-code] (CR) : Elaborates on the command error message '[error-message-code]' for many commands; use `man explain' for a complete list.

Cray Job Accounting (ja) Command:

ja (CR)
{[}executable] (CR)
ja -csf (CR) : This command sequence enables Job Accounting storing the information in a file of the form `.jacct[jobid]', with options `c' giving a command report, `f' giving a command flow report, `s' giving a multitasking breakdown summary report. Note that the NPACI service unit charges are approximately one cpu hour on the T90 and one element hour on the T3E, assuming average memory (about 16MW) usage. Caution: In general, parallel processing on the YMP series like the T90 is very expensive.

Cray Perftrace (perf) or Performance Trace Facilities:

f90 -ef [source].f (CR)
segldr -l perf [source].o (CR)
a.out > [source].perf (CR)
segldr - l perf perf[n] [source].o (CR)
a.out >> [source].perf (CR) : Compiles the FORTRAN 77 program `[source].f' for use for the Cray Perftrace or Performance Trace facilities. (Flowtrace results are similarly found in the output of the executable file executed after loader statement.) The library suboption here is `perf' for referencing the libperf.a library, which has several levels, where `[n]' is the level `', `1', `2' or `3'.

Cray Hardware Performance Monitor (hpm):

hpm -g[n] -d [executable] > [source].hpm[n] (CR) : Simulates the Hardware Performance Monitor with `[executable]' and level `l' = `0' (scalar activity), `1' (hold issue conditions), `2' (memory use), or `3' (instruction and vector operations). The option `-d' means that a dedicated machine is simulated.

Cray JumpTrace (jt) and JumpView (jumpview): JumpTrace and JumpView help gather performance statistics in the form of a report. Some use examples are:

Fortran Example:

f90 -ef [pgm].f
jt ./a.out
jumpview


C Example:

cc -ltrace -Gp [cpgm].c
jt ./a.out
jumpview -Luch >[cpgm].listing


JumpView Main Menu:

----------------------------------------------------- MAIN MENU
1  Master Summary          |  7  List by Average Time/Call
2  Routines: List by Time  |  8  Operating Environment
3  List by Megaflops       |  9  Long Report by Routine Name
4  List by In-Line Factor  | 10  Detail Report by Symbol
5  List by Name            | 11  Detail Report by Block
6  List by Calls           | 12  Options
----------------------------------------
  H  HELP
  Q  QUIT
          Enter Number/Letter of Action Desired
---------------------------------------------------------------


Cray Autotasking Expert Performance System (atexpert):

atexpert [options] (CR) : Autotasking expert performance system, needing X-windows display for full power. See also `atchop' and `atscope'.

make [-options] [step-name] (CR) : Makes the files [files] according to the template in the `makefile'. E.g., the file `makefile.unicos_2':

# Use ``make -f make.unicos_2 mrun>& pgm.l &;
run<data>out''.
SOURCES = pgm.f
OBJECTS = pgm.o
FLAGS = -em    
mrun : $(OBJECTS) 
segldr -o run $(OBJECTS) 
                                         
.f.o : f90  $(FLAGS)  $*.f

{CAUTION: The commands, like `segldr' or `f90', must be preceded by a `Tab-key' tab as a delimiter, but the tab will not be visible in the UNIX listing.}
fmgen -m [make-name] -c f90 -f [-flag] -o [executable] [source].f (CR) : Automatically generates a makefile for compiling under the `f90' compiler and loading up the executable file named `[executable]'. Invoke with `make -f [make-name] [executable](CR)' and the execute `[executable]'. Also produces steps for profiling, flow-traces, performance traces, and clean-up, in the heavily documented makefile. For example, `make -c f90 -f -r3 -o run pgm.f (CR)' produces a makefile named `makefile', executable named `run', an information listing named `[name in program statement].l' with loops marked by optimization type, etc.; the making is done with `make run (CR)'. Caution: the makefile only uses the source name only when that coincides with the name used in the Fortran `program' statement and only one type of `f90' flag can be used. These flaws can be corrected by editing the resulting makefile `[make-name]'.

mailx (CR) : Shows user`s mail; caution: `mailx' is close to the usual Unix mail, whereas the UNICOS `mail' command is NOT; use the subcommand `t [N](CR)' to list message number `[N]' , `s [N] mbox (CR)' to append message `[N]' to your mailbox `mbox' file or `s [N] [file](CR)' to append `[N]' to another file; `e [N] (CR)' to edit number [N] or look at a long file with `ex' {see Section on `EX' below}; `v [N] (CR)' to edit number [N] or look at a long file with `vi'; `d [N] (CR)' deletes {your own mail!} `[N]'; `m [user] (CR)' permits you to send mail to another account `[user]'; a `~m [N] (CR)' inside the message after entering a subject, permits you to forward message `[N]' to `[user]', `\d (CR)' to end the new message {see the send form below;`x' quits `mailx' without deleting {use this when you run into problems}; and `q (CR)' to quit.
mailx [user] (CR) : Sends mail to user `[user]'; the text is entered immediately in the current blank space; carriage return to enter each line; enter a file with a `~r[filename] (CR)'; route a copy to user `[userid]' by `~c[userid] (CR)'; enter the `ex' line editor with `~e (CR)' or `vi' visual editor with `~v (CR)' (see Sections on EX and on VI) to make changes on entered lines, exiting `ex' with a `wq (CR)' or `vi' with a `:wq' (CR)'; exit `mailx' by entering `\d (CR)'. {A bug in the current version of Telnet does not allow you to send a copy using the `cc:' entry.
mailx [name]@[machine].[dept].uic.edu < [filename] (CR) : Sends the UNICOS file `[filename]' to user `[name]' on some UNIX or other machine.

qsub [options] (CR) : Submit a batch job to the queue; see `man qsub (CR)' for more information. The option, for example, `-lM [16Mw]' permits running jobs with up to 16 mega words of memory, for example. The option `[myjob].script' provides the script instructions for running a background job. Note that NPACI users must specify a script line

#QSUB -lM [memory-amount]
specifies a memory of `[memory-amount]' bytes for a job using `Mw' to denote mega words, instead of an option of `qsub'; and also required is

#QSUB -lT [CPU-time-amount]
specifying the amount of wall (user plus system) clock time in seconds. In addition, T3E users must also specify

#QSUB -l mpp_p=[t3e_procs],mpp_t=[t3e_time]
giving the number of T3E processors and time on the T3E; and also

#QSUB -q mpp
giving the T3E queue name `mpp' (Caution: you must be in the `mpp' group to use this queue, but you can check it by the command

grep [username] /etc/group (CR)
on the T90, whereas the default queue is `batch'.

For more information about batch processing with NQS, click on:


Using the CRAY T90
Using the CRAY T3E

qstat [options] (CR) : Display status of queued batch jobs; see `man qsub (CR)' for more information.
/mpp/bin/mppstat (CR) : Not an NQS command, but displays the current T3E configuration and the number of available processors (PEs).
/usr/local/adm/access/bin/qstatmpp (CR) : Not an NQS command, but displays the currently queued T3E jobs.

For optimization, it is recommended that your f90 program aid the f90 vector model, i.e. structure the code so that the compiler can automatically recognize as vectorizable. Usually only inner most loop is vectorizable. Avoid loop GOTOs and IFs. Avoid CALLs within loops. Avoid loop READs and WRITEs. Use vectorizable functions. Avoid data dependencies. Use compiler directives, such as `!DIR$ VECTOR' and `!DIR$ NOVECTOR'. Minimize vector strides. Tune code to Fortran column-wise environment in the physically linear memory. Don't even think about using tabs, except in makefiles.

See also Section ``Execution of Cray T90 Fortran90 (f90)'' and Subsection ``T90 UNICOS f90 Compile, Load and Execution Commands''. Also see the appropriate sections, `docview' and `man cc' for items on Cray Standard C.

``FORTRAN90 Array Notation'' {f90 allows Fortran90 extensions for array, making array statements like `AS =S', `C = A +B', `A(1:50) = B(1:100:2)' for appropriately dimensioned arrays AS, A, B and C, and scalar S (i.e., like AS(i,j) = S, for all i and j within subscript bounds); in general 'A([start]:[end]:[step])' references the single subscript array section for i = [start] to [end] in steps of [step]. Other examples are `a(i,:)' for the i-th row of array `a', `a(:,j)' for the j-th column, `a(1::2)' for the odd vector elements, `a(n:1:-1)' for the `n' vector elements of `a' in reverse order, and `z(1:n) = -log(z(1:n))' or `z(1:n) = ranf()'.}
real [variables-list] {The f90 `real' declaration declares variables and array elements as 32-bit (4-byte) words with only 23-bits allotted to the fraction for IEEE precision. This is somewhat different from the old non IEEE precision Cray where real meant an 8 byte or 64 bit real. Thus in f90 code, use the built-in functions `abs', `sqrt', `exp', `amax1' and so forth. The IEEE precision f90 `double precision' declaration is 64-bit with a 54-bit fraction, and hence is entirely different from old Non-IEEE precision Cray `double precision'.}
POINTER (P,A) {The f90 `pointer' statement declares that the declared integer (usually) variable holds (points to, for C-fans) the shifted initial (base) address of the declared array A.}
``Execution Time Allocation'' {f90 allows execution time storage of temporary arrays within subprograms, rather than at compile time; means that f90 will be less sensitive to array bounds over-runs.}
open ([unit],file=`[fn]',status='unknown') {Format of f90 OPEN statement assigning unit number [unit] to filename [fn]; place in program after declarations; [unit] = 5 defaults to UNIX `stdin' as does [unit] = * for read statements or reads from the terminal unless it is redirected by an `open' or a `lt;'; [unit] = 6 defaults to UNIX `stdout' as does [unit] = * for write statements or writes to the terminal unless it is redirected by an `open' or a `>'; [unit] = 0 defaults to UNIX `stderr' or writes diagnostics to the terminal unless it is redirected by an `open' or a `>&'; note that file names are placed in quotes in the OPEN statement; see also `man' for UNICOS `assign' and `env' statements. }
save [variable or array name list separated by commas] {The save statement is essential in f90 subroutines to save parameter variable values for later calls to a subroutine; the `-ev' option of f90 provides a better solution to this problem; if not used can lead logic errors, especially for users accustomed to F66 Fortran in which variables are saved after the RETURN statement is executed, but lost in f90.}
recursive [function or subroutine]([subprogram arguments]) {The 'recursive' prefix is required on subprograms called recursively, but also the recursive suboption is needed in the compiler statement.}
[statement] ! [embedded comment] {The line embedded comment is now legal in Cray Fortran.}
intrinsic [f90-function1][,[f90-function2]] {An Intrinsic function is needed in `f90' to declare any Fortran90 intrinsics, such as ANY, DOT_PRODUCT, MAXVAL, RESHAPE, ALL, EOSHIFT, MINLOC, SPREAD, COUNT, FLOAT, MINVAL, SUM, CSHIFT, MATMUL, PACK, TRANSPOSE, MAXLOC, PRODUCT, UNPACK.}

PACK([array],[mask-array][,[vector]]) {Transforms (packs) the array `[array]' into a vector `[vector]' (an optional argument, which if not present, the output goes to the value of the function) according to the true values of the `[array]'-conformable, logical mask `[mask-array]'. }
UNPACK([vector],[mask-array],[field-array]) {Transforms (unpacks) the vector `[vector]' into the array `[field-array]' according to the true values of the `[field-array]'-conformable, logical mask `[mask-array]'. }
SPREAD([array],[dim],[ncopies]) {Transforms (spreads) the source array `[array]' into the output value of the function with `[ncopies]' copies along the dimension `[dim]' (horizontal copies if `[dim]'=1 and vertical if `[dim]'=2. }
RESHAPE([array],[shape][,[pad]][,[order]]) {Transforms (reshapes) the source array `[array]' into the output value of the function with shape `[shape]' with order `[order]' padding the array `[pad]'. }

The reduction functions reduce the input to a scalar output.

SUM([array][,[dim][,[mask]]]) {The `SUM' function computes the sum of the elements of the array `[array]' along the dimension `[dim]' (by columns if `[dim]'=1 or by rows if `[dim]'=2) according to the true values in the conditional mask `[mask]', if present. This function makes the Cray sum function the same as the Connection Machine version. }
PRODUCT([array][,[dim][,[mask]]]) {The `PRODUCT' function computes the product of the elements of the array `[array]' along the dimension `[dim]' (by columns if `[dim]'=1 or by rows if `[dim]'=2) according to the true values in the conditional mask `[mask]', if present. }
MAXVAL([array][,[dim][,[mask]]]) {The `MAXVAL' function computes the maximum value of the elements of the array `[array]' along the dimension `[dim]' (by columns if `[dim]'=1 or by rows if `[dim]'=2) according to the true values in the conditional mask `[mask]', if present. }
MINVAL([array][,[dim][,[mask]]]) {The `MINVAL' function computes the minimum value of the elements of the array `[array]' along the dimension `[dim]' (by columns if `[dim]'=1 or by rows if `[dim]'=2) according to the true values in the conditional mask `[mask]', if present. }
COUNT([mask][,[dim]]) {The `COUNT' function computes the number of the true elements of the logical array `[mask]' along the dimension `[dim]' (by columns if `[dim]'=1 or by rows if `[dim]'=2), if present. }
ANY([mask][,[dim]]) {The `ANY' function computes if there are any true elements in the logical array `[mask]' along the dimension `[dim]' (by columns if `[dim]'=1 or by rows if `[dim]'=2), if present, and returns a logical true or false answer. }
ALL([mask][,[dim]]) {The `ALL' function computes if there are all true elements in the logical array `[mask]' along the dimension `[dim]' (by columns if `[dim]'=1 or by rows if `[dim]'=2), if present, and returns a logical true or false answer. }

The manipulation functions rearrange the elements of the target matrix.

TRANSPOSE([array]) {The `TRANSPOSE' function transposes the 2-subscript array `[array]' with the result array of reversed dimensions. }
EOSHIFT([array],[shift][,[boundary][,[dim]]]) {The `EOSHIFT' function does an end-off shift on the array `[array]' along the dimension `[dim]' using the boundary value(s) `[boundary]' to fill in, if necessary. Caution: Connection Machine arguments have a different order. }
CSHIFT([array],[shift][,[boundary][,[dim]]]) {The `CSHIFT' function does a circular shift on the array `[array]' along the dimension `[dim]' using the boundary value(s) `[boundary]' to fill in, if necessary. Caution: Connection Machine arguments have a different order. }

The location functions find the location of elements of the target matrix.

MAXLOC([array][,[mask]]) {The `MAXLOC' function finds the first element of target array `[array]' having the maximum value, relative to the conditional mask `[mask]', if present. }
MINLOC([array][,[mask]]) {The `MINLOC' function finds the first element of target array `[array]' having the minimum value, relative to the conditional mask `[mask]', if present. }

The matrix multiply functions compute the matrix products of the target matrices.

MATMUL([array1][array2]) {The `MATMUL' function computes the matrix product of target arrays `[array1]' and `[array2]' commensurate for multiplication, with the result matrix of appropriate size. This function is also used for matrix-vector multiplication. }
DOT_PRODUCT([vector1][vector2]) {The `DOT_PRODUCT' function computes the scalar, dot product of target vectors `[vector1]' and `[vector2]', with the scalar result. Caution: the Connection Machine function is `dotproduct'. }

T90 Fortran90 (f90) Differences:

The following f90 code contains examples of use of many of the Fortran90 array intrinsic functions mentioned above. There are some rules:

1. Intrinsic statement is needed for all f90 intrinsics within f90 codes.
2. Constructors of the form b=(/1 2 3/) work with the f90 compiler.
3. Fortran90 array intrinsics used within f90 will take no auxiliary markers or keywords like "dim=" or "mask=".
4. array sections can not be used in print statements: NOT print*,b(1:3)
5. How do you sum an entire array only subject to a mask, but with no dimension restrictions?

       If  b =  1  3  5            logical mask=b.gt.3
                2  4  6
   
       then   s3=sum(b,1,mask)  or  s2=sum(b,2,mask) work when real s3(3),s2(2)
   
       but    isum=sum(b,mask)  or  isum=sum(b,,mask) or isum=sum(b,:,mask)
              do NOT work.
       That is how do I enter a scalar dim for the whole array?
   

   f90 -O3 -r3 -o run pgm.f&
   run>&pgm.out&
%%%%%%%%%%% pgm.f=t90f90test.f %%%%%%%%%
      program f90test
code98:  compare ranf() and random_number pseudo random number generators
code97:  update by removing old comments to cmfortran
code96:  retest=f90test.f redone on borg = convex spp1200/xa-16
      integer, parameter :: m = 6
      integer, parameter :: n = 4
      integer :: i,j
      integer, dimension(2) :: s2, ctr1, ctr2, ctr3, b2
      integer, dimension(3) :: s3 ,at ,ar1 ,ar2 ,br1 ,br2
      integer, dimension(4) :: as(4)
      integer, dimension(2,2) :: c ,bi
      integer, dimension(2,3) :: b, a
      integer, dimension(3,2) :: ct
      integer, dimension(3,4) :: cs
      integer, dimension(4,3) :: cst
      logical, dimension(2,3) :: test
      logical, dimension(64,64) :: inmask
      real, parameter :: tol = 0.5e-5
      integer, parameter :: niter = 5000
      real :: diffav
      real, dimension(8,8) :: us
      real, dimension(64,64) :: u , du
      real :: ranf, xran
      real, dimension(m,n) :: uniranf, uniran
      real, dimension(n,m) :: truniranf, truniran
      intrinsic  sum,maxval,minval,product
     & ,dot_product,matmul,transpose
     & ,cshift,eoshift,spread
      data b/1,2,3,4,5,6/     !replace constructors initialization
      data as/2,3,4,5/
      data at/2,3,4/
c --------------------Array Constructors:
       b(1,1:3) = (/1, 3, 5/)  ! initialize first row, along dimension 2.
       b(2,1:3) = (/2, 4, 6/)  ! initialize second row, along dimension 2.
      print*,'Note: constructors like "(/1,2/)" allowed in fc9.5'
      br1 = b(1,:)
      br2 = b(2,:)
      print60,br1,br2
60    format(' b(2,3)'/(3i3))
c --------------------Sum Function sum:
      isum = sum(b) ! => isum = 21; i.e., Front-End scalar.
      print61,' isum=sum(b)=',isum
61    format(1x,a36,i4)
      isum = sum(b(:,1:3:2)) ! => isum = 14; sole ':' means all values '1:2'.
      print61,' isum = sum("b(:,1:3:2)")=',isum
      bi=b(:,1:3:2)
      isum=sum(bi)
      print61,' isum = sum("b(:,1:3:2)")=',isum
      print*,'CAUTION: "dim=", etc., markers= NOT allowed in intrinsics'
      s2 = sum(b,2) ! redeclared with the correct array section shape.
      print62,' s2 = sum(b,2)=',s2  ! => s2 = (/9,12/), row sums
62    format(1x,a32,2i3)
      s3 = sum(b,1)  ! => s3 = (/3,7,11/); column sums.
      print63,' s3 = sum(b,1)=',s3
63    format(1x,a32,3i3)
      print*,'CAUTION:  "mask=" marker= STILL not allowed either.'
      s3 = sum(b,1,b.gt.3) ! => s3 = (/0,4,11/); i.e., conditional col sum
      print63,' s3 = sum(b,1,"b.gt.3") =',s3  
      test=b.gt.3
      s3 = sum(b,1,test) ! => s3 = (/0,4,11/); i.e., conditional col sum
      print63,' s3 = sum(b,1,"b.gt.3") =',s3  
      s2 = sum(b,2,test) ! => s2 = (/5,10/); i.e., conditional row sum
      print62,' s2 = sum(b,2,b.gt.3) =',s2  
cf8er:isum = sum(b,0,test) ! => isum = 18; i.e., add only elements
cf8er:print61,' isum = sum(b,0,b.gt.3) =',isum ! that are greater than three.
      print*,' CAUTION:  If "sum(array[dim[,mask]])", CANT use zero (0)'
     &      ,' for [dim] for whole array when there is a mask.'
c --------------------Maximum Value Function maxval:
      imax = maxval(b) ! => imax = 6; array maximum value.
      print61,' imax = maxval(b)=',imax
      s3 = maxval(b,1) ! => s3 = (/2,4,6/); column maximums.
      print63,' s3 = maxval(b,1)=',s3
      s2 = maxval(b,2) ! => s2 = (/5,6/); row maximums.
      print62,' s2 = maxval(b,2)=',s2
c --------------------Minimum Value Function minval:
      imin = minval(b) ! => imin = 1; array minimum value.
      print61,' imin = minval(b)=',imin
c --------------------Product Function product:
      s2 = product(b,2) ! => s2 = (/15,48/); products of column elements.
      print62,' s2 = product(b,2)=',s2
c --------------------Dot Product Function dot_product:
      idot = dot_product(br1,br2) ! => idot = 44; dot product of row
      print61,' idot = dot_product(b(1,:),b(2,:))=',idot ! vectors of b.
      print*,' CAUTION:  Array syntax not allowed in actual arguments.'
c --------------------Matrix Multiplication Function matmul:
      ! assuming array b of the previous section.
      ![Ans] = matmul([Array_1],[Array_2]) ! computes matrix multiplication
                                           ! of two rank two matrices.
      c = matmul(b(:,1:2),b(:,2:3)) ! => c(1,:)=(/15,23/);c(2,:)=(/22,34/).
      c=transpose(c)
      print623,'c=matmul(b(:,1:2),b(:,2:3))=',c
623   format(1x,a36/(2i3))
      ![Ans] = transpose([Array]) ! transforms an array to its transpose.
      ct = transpose(b) ! => ct(1,:)=(/1,2/);ct(2,:)=(/3,4/);ct(3,:)=(/5,6/).
      ctr1 = ct(1,:)
      ctr2 = ct(2,:)
      ctr3 = ct(3,:)
      print623,'ct = transpose(b)=',ctr1,ctr2,ctr3
c --------------------Circular Shift Function cshift:
        ! assume b is again initialized as
        !        b =  1 3 5
        !             2 4 6
      a = cshift(b,1,2)  ! => a = 3 5 1
                         !        4 6 2
cshift  EG1:
      ar1 = a(1,:)
      ar2 = a(2,:)
      print633,'a = cshift(a,1,2)=',ar1,ar2
633   format(1x,a36/(3i3))
    ! i.e., b(i,(j+shift) "mod" n) -> a(i,j) for j=1:2, etc.;
    ! nonstandard modulus fn: 0 "mod" n = n; 1 "mod" n = 1; ...;  n "mod" n = n
    ! i.e., the result is computed from shifting subscript in specified
        ! dimension of the source array by the specified shift.
      a = cshift(b,-1,2)  ! => a = 5 1 3
                          !        6 2 4
cshift  EG2:
      ar1 = a(1,:)
      ar2 = a(2,:)
      print633,'a = cshift(b,-1,2)=',ar1,ar2
        ! i.e., b(i,(j+shift) "mod" n) -> a(i,j) for j=2:3, etc.
cshift  EG3:
      s2(1) = 1
      s2(2) = 2
      a = cshift(b,s2,2)  ! a = 3 5 1
                          !     6 2 4
        ! i.e., an array-valued shift, or shift per row.
      ar1 = a(1,:)
      ar2 = a(2,:)
      print633,'a = cshift(b,(/1,2/),2)=',ar1,ar2
cshift Laplace Example:
        ! Jacobi Iteration for a 5-star discretization of 
        !        2D Laplace's equation:
      u = 0
      u(1,:)=2
      u(64,:)=2
      u(:,1)=2
      u(:,64)=1
      inmask = .FALSE.
      inmask(2:63,2:63) = .TRUE.
      diffav = 1
      iter=0
      do while (diffav.gt.tol.and.iter.lt.niter)
         iter=iter+1
         du = 0
         where(inmask)
            du = 0.25*(cshift(u,1,1)+cshift(u,-1,1)+cshift(u,1,2)
     &          +cshift(u,-1,2)) - u
            u = u + du
         end where
         du = du*du
         diffav = sqrt(sum(du)/(62*62))
      end do 
        ! which is the main program fragment of laplace.fcm.
      print*,'CAUTION: array sections not allowed in print'
      us = u(1:64:9,1:64:9)
      us=transpose(us)
      print66,'u = laplace-shift(u)= ; iter=',iter,'; av-diff ='
     &       ,diffav,us
66    format(1x,a36,i5,a11,e10.3/(8f8.4))
c --------------------End Off Shift Function eoshift:
      a = eoshift(b,-1,0,1) ! a = 0 0 0 note default boundary value is 0.
                            !     1 3 5
      ar1 = a(1,:)
      ar2 = a(2,:)
      print633,'a = eoshift(b,-1,0,1)=',ar1,ar2
      s2=(/-1,0/)
      b2=(/7,8/)
      a = eoshift(b,s2,b2,2) ! => a = 7 1 3
                             !        2 4 6
      ar1 = a(1,:)
      ar2 = a(2,:)
      print633,'a = eoshift(b,(/-1,0/),(/7,8/),2)=',ar1,ar2
      a = eoshift(b,2,0,2) ! => a = 5 0 0
                           ! =>     6 0 0
      ar1 = a(1,:)
      ar2 = a(2,:)
      print623,'a = eoshift(b,2,2)=',ar1,ar2
c --------------------Spread Function spread:
      cs = spread(as,1,3)
         ! contents of cs:
         !        2 3 4 5
         !        2 3 4 5
         !        2 3 4 5
      cst = transpose(cs)
      print64,'as =',as
64    format(1x,a32,4i3)
      print643,'cs = spread(as,1,3)=',cst
643   format(1x,a36/(4i3))
c --------------------
      cs = spread(at,2,4)
         ! contents of c:
         !        2 2 2 2
         !        3 3 3 3
         !        4 4 4 4
      cst = transpose(cs)
      print63,'at =',at
      print643,'cs = spread(at,2,4)=',cst
c ---------------------------------------------------------------------------
! i.e., b=spread(a,d,c)  =>
! a(n_1,n_2,...,n_(d-1),n_d,...,n_r) -> b(n_1,n_2,...,n_(d-1),c,n_d,...,n_r)
! where r is the rank of source array a and n_i is the size of dimension i;
! noting that a new dimension of size c is added before dimension d.
c ---------------------------------------------------------------------------
! Initialize scalar xran with a pseudo random number
      call random_number(harvest=xran)
      call random_number(uniran)
! xran and uniran contain uniformly distributed random numbers
      truniran = transpose(uniran)
      write(6,65) xran, truniran
65    format(' f90 uniform random_number(): xran =',f14.10/ 
     &   ' and f90 subroutine random_number() uniform random array:'
     &         /(4f14.10))
! standard UNICOS random number generator ranf:
      do i = 1, m
         do j = 1, n
            uniranf(i,j) = ranf()
           enddo
      enddo
      truniranf = transpose(uniranf)
      write(6,651) truniranf
651   format(' UNICOS function ranf() uniform random array:'/(4f14.10))
      stop
      end
%%%%%%%%%%% end pgm.f=t90f90test.f %%%%%%%%%

• Click here to get `t90f90test.f', T90 Fortran 90 code

%%%%%%%%%%% begin pgm.output = t90f90test.output %%%%%%%%%
 Note: constructors like "(/1,2/)" allowed in fc9.5
 b(2,3)
  1  3  5
  2  4  6
                         isum=sum(b)=  21
            isum = sum("b(:,1:3:2)")=  14
            isum = sum("b(:,1:3:2)")=  14
 CAUTION: "dim=", etc., markers= NOT allowed in intrinsics
                   s2 = sum(b,2)=  9 12
                   s3 = sum(b,1)=  3  7 11
 CAUTION:  "mask=" marker= STILL not allowed either.
         s3 = sum(b,1,"b.gt.3") =  0  4 11
         s3 = sum(b,1,"b.gt.3") =  0  4 11
           s2 = sum(b,2,b.gt.3) =  5 10
  CAUTION:  If "sum(array[dim[,mask]])", CANT use zero (0) for [dim] for whole array when there is a mask.
                    imax = maxval(b)=   6
                s3 = maxval(b,1)=  2  4  6
                s2 = maxval(b,2)=  5  6
                    imin = minval(b)=   1
               s2 = product(b,2)= 15 48
   idot = dot_product(b(1,:),b(2,:))=  44
  CAUTION:  Array syntax not allowed in actual arguments.
         c=matmul(b(:,1:2),b(:,2:3))=
 15 23
 22 34
                   ct = transpose(b)=
  1  2
  3  4
  5  6
                   a = cshift(a,1,2)=
  3  5  1
  4  6  2
                  a = cshift(b,-1,2)=
  5  1  3
  6  2  4
             a = cshift(b,(/1,2/),2)=
  3  5  1
  6  2  4
 CAUTION: array sections not allowed in print
        u = laplace-shift(u)= ; iter= 4730; av-diff = 0.499E-05
  2.0000  2.0000  2.0000  2.0000  2.0000  2.0000  2.0000  1.0000
  2.0000  1.9762  1.9479  1.9090  1.8491  1.7440  1.5208  1.0000
  2.0000  1.9573  1.9068  1.8387  1.7387  1.5836  1.3402  1.0000
  2.0000  1.9469  1.8844  1.8014  1.6836  1.5141  1.2817  1.0000
  2.0000  1.9469  1.8844  1.8014  1.6836  1.5141  1.2817  1.0000
  2.0000  1.9573  1.9068  1.8387  1.7387  1.5836  1.3402  1.0000
  2.0000  1.9762  1.9479  1.9090  1.8491  1.7440  1.5208  1.0000
  2.0000  2.0000  2.0000  2.0000  2.0000  2.0000  2.0000  1.0000
               a = eoshift(b,-1,0,1)=
  0  0  0
  1  3  5
   a = eoshift(b,(/-1,0/),(/7,8/),2)=
  7  1  3
  2  4  6
                  a = eoshift(b,2,2)=
  5  0
  0  6
  0  0
                             as =  2  3  4  5
                 cs = spread(as,1,3)=
  2  3  4  5
  2  3  4  5
  2  3  4  5
                             at =  2  3  4
                 cs = spread(at,2,4)=
  2  2  2  2
  3  3  3  3
  4  4  4  4
 f90 uniform random_number(): xran =  0.5801136486
 and f90 subroutine random_number() uniform random array:
  0.9505127350  0.3056509439  0.0986253383  0.6938844384
  0.7863714253  0.6891007107  0.2765484551  0.9344770142
  0.2976202640  0.3826622387  0.6204460278  0.2120929553
  0.4536999003  0.1329027055  0.0835029668  0.1306527482
  0.0062619416  0.8318579032  0.9903771206  0.8625969805
  0.2757364264  0.5829797958  0.9793469434  0.8189092940
 UNICOS function ranf() uniform random array:
  0.5407187129  0.0187994091  0.3141160167  0.7651821004
  0.9415271082  0.2893071356  0.5849975196  0.9030257778
  0.8866798463  0.4966670053  0.3964840582  0.8718218141
  0.9311052262  0.5954839343  0.2096123584  0.8881281192
  0.4641396487  0.6280308383  0.4467249313  0.4578495774
  0.2349011311  0.7635970977  0.5911920675  0.4438340178
 STOP   executed at line 222 in Fortran routine 'F90TEST'
 CPU: 1.827s,  Wallclock: 0.533s,  24.5% of 14-CPU Machine
 Memory HWM: 308988, Stack HWM: 37805, Stack segment expansions: 0
%%%%%%%%%%% end pgm.output = t90f90test.output %%%%%%%%%

• Click here to get `t90f90test.output', corresponding T90 Fortran 90 output

Cray T3E f90 Differences:

Here is a sample code with many examples, heavily commented and followed by the actual output run on t3e.npaci.edu using the commands

   f90 -O3 -r3 -Xm -o fpgm fpgm.f &
   mpprun -n1 fpgm  >& fpgm.output &

%%%%%%%%%%% pgm.f=cf97test.f %%%%%%%%%

• Click here to get `t3e97test.f', T90 Fortran 90 code
• Click here to get `t3e97test.output', corresponding T90 Fortran 90 output

[variable] = ssum (n,a(m),k) {The optimized scientific library SCILIB function `ssum' computes the sum of `n' elements of array `a' starting from element `m' in steps of `k'; the equivalent but not optimal, do loop is {sum=0; do 1 i=m,n+n-1,k; 1 sum=sum+a(i)}; e.g. `sum=ssum(n,a,1)' returns the sum of the first `n' elements of the array 'a'; if `a' is an m X n 2-subscript array, use `t = ssum(m*n,a(1,1),1)'; use `man ssum' for more information; the `-l libsci.a' option in the `segldr' should be optional. UPDATE: In cf77 version 6.0, `ssum' has been replaced by the Fortran90 `sum' function.}
[variable] = sdot (n,a,1,b,1) {The optimized SCILIB function `sdot' returns the calculated value of the dot product of `n' elements of the vectors `a' and `b' in steps of 1; the `-l libsci.a' option of the `segldr' should be optional. UPDATE: In cf77 version 6.0, `sdot' has been replaced by the Fortran90 `DOT_PRODUCT' function.}
call mxm (a,m,b,kmax,c,n) {The optimized SCILIB subroutine returns the calculated value of the full matrix by matrix product of the `m X kmax' array `a' and the `kmax X n' array `b' into the `m X n' output array `c'; use `mxma' for multiplication of sub-matrices when the matrices are not full; use `man mxm' (ignore UNICOS function of the same name) or 'man mxma' for more information; the `-l libsci.a' option of the `segldr' should be optional. UPDATE: In cf77 version 6.0, `mxm' has been replaced by the Fortran90 `MATMUL' function.}
call mxv (a,m,b,n,c) {The optimized SCILIB subroutine returns the calculated value of the full matrix by vector product of the `m X n' array `a' and the `n' vector `b' into the `m X n' output vector `c', by rolling up the `j' loop; use `man mxv'; the `-l libsci.a' option of the `segldr' should be optional. UPDATE: In the T90 cf77 version 6.0, `mxv' has been replaced by the Fortran90 `MATMUL' function.}
call random_number([HARVEST=][variable]) {F90 Pseudo-random number generator on [0,1], as intrinsic subroutine rather than intrinsic function, that gets the first or next random number or array a stores it in the user output variable or array `[variable]'. For example:

real s, r(100,100)
call random_number(harvest=s)
call random_number(r)

See `man random_number' for more information, or `man rand_seed' for changing the random sequence. }
[random-variable] = ranf() {UNICOS Pseudo-random number generator on [0,1] that gets the first or next random number, e.g.,

real s, r(100,100)
s = ranf()
do i = 1,100
   do j = 1,100
      r(i,j) = ranf()
   enddo
enddo

or use `r(1:n) = ranf()' in Fortran90 notation; use `x(1:n) = -log(r(1:n))' to convert to an exponential distribution; change the random generator seed using `call ranset([new-seed]), but it is not necessary to start with a seed; `ranf' is a great random number generator since it properly vectorizes in loops; use `man ranf' for more information, including use for C/C++ as `_ranf()' which requires the following include and declaration statements:

#include 
double _ranf(void);

}
call wheni[reln] ([nfind],[iarray],[inc],[itarget],[index],[nval]) (CR) {Finds all integer array (`[iarray]') elements in relation (`[reln]') to the integer target (`[itarget]'); `[reln]' = `lt', `le', `gt' or `ge'; `[n]' is the number of elements to be searched in increments of `[inc]'; `[index]' is the integer array of the indices of the output; and `[nval]' is the number in indices found.}

These statements are placed in the Fortran source just before the loop or other entity they are to effect, but they stay in effect until the opposite directive is given. However, for every toggling compiler directive that turns some action on, there is another directive with an `NO' prefix appended that turns that action off. The leading `C' must be in column 1 and a blank must be in column 6. For more information, see F90 Vol. 1: Fortran Reference Manual, Sect. 1.6 Compiler Directives.

!DIR$ VECTOR {Compiler directive causes all following inner DO loops to be vectorized unless the loop is known to have only one iteration and until superceded by another directive that alters vectorization.}
!DIR$ NOVECTOR {Directive turns off vectorization at next DO loop until turned back on.}
!DIR$ VSEARCH {Directive permits optimization of loops that can have a premature exit, as with convergence of an iteration. !DIR$ NOVSEARCH directive turns it off.}
!DIR$ INLINE {Directive turns on inlining, inline code generation, of subprograms if `-I' or `-o inline' f90 options are used; !DIR$ NOINLINE turns inlining off.}

These directives effect scalar optimization at the point at which the directive appears and only affects the local program unit, such as the loop it appears in.

!DIR$ BL {Directive turns on bottom loading for loops, pre-fetching data for the next loop iteration; !DIR$ NOBL turns BL off.}
!DIR$ NOSIDEEFFECTS [subprogram-name] {Allows keeping data in registers across subprograms, if no global data (i.e., arguments of common blocks) are changed.}
!DIR$ SUPPRESS [variable-list] {Directive temporarily suppresses scalar optimization on variables in loops containing the directive.}

These compiler directives hold only for the loop immediately following the directive.

!DIR$ IVDEP {Directive causes compiler to Ignore Vector DEPendencies in only the next inner most DO. Disabled by the NOVECTOR directive.}
!DIR$ NEXTSCALAR {Directive causes only the very NEXT DO loop to be executed in SCALAR Mode with vectorization resuming if on. Disabled by the NOVECTOR directive.}
!DIR$ SHORTLOOP {Directive reduces vectorization overhead for the very next loop, presumed SHORT or has less than 64 iterations. Disabled by the NOVECTOR directive.}
!DIR$ RECURRENCE {Directive turns on vectorization of reduction loops (e.g., sum loops); !DIR$ NORECURRENCE turns it off. Disabled by the NOVECTOR directive.}

These compiler directives alter the way memory is handled.

!DIR$ VFUNCTION [external-function-list] {Directive declares Vector version of an external CAL FUNCTION, where CAL is the Cray Assembler Language, but the function can not be declared in an External statement; works with list of CAL functions separated by commas. See the f90 Vol. 1 for other restrictions. }
!DIR$ AUXILIARY [array-list] {Storage directive allows assignment to the secondary disk storage for the Cray Y-MP only.}

These are used the same way as the vector directives.

!DIR$ BOUNDS [array-names] {Allows the checking of array subscript bounds, but inhibits vectorization; applies to all arrays unless particular one are listed as arguments.}
!DIR$ NOBOUNDS {Prevents checking of subscript bounds.}
!DIR$ FLOW {Turns on Flow-trace and `!DIR$ NOFLOW' turns it off.}

For more information on compiler directives and other f90 statements, refer to the `Cray Fortran (CFT) REFERENCE MANUAL'. Addition information on SCILIB functions can be found in the Cray Library Reference Manual, a copy of which is found in the UIC Supercomputing Support Office along with many other manuals.

The Cray supercomputers now have parallelization or tasking features in additions to vectorization features. However, the cost of running Cray Fortran is extremely large, because the user is charged for time on all processors utilized. In contrast, the user is not charged for each vector element with vectorization. HENCE THE USER SHOULD ONLY USE THE MULTITASKING FEATURES WHEN ABSOLUTELY NECESSARY.} Macrotasking refers to large grain or subroutine level parallelization. Microtasking refers to parallel loop optimization through compiler directives. Autotasking refers to automatic microtasking by the Fortran Preprocessor `fpp', i.e., through automatic code generation for multitasking. Compiler f90, preprocessor fpp and mid-processor fmp are currently version 4.0 at NPACI. More information is found on the NPACI Cray T90, in the directory `/usr/local/doc' files or subdirectories such as `cf77_50.rn' release notes or `unicos.7.0' sections.

A typical job accounting execution sequence might be
ja (CR)
${TMP}/[fn] < [data] > & [output] & (CR)
with the job accounting information appearing in a file of the form `.jacct[jobid]'. Including the pass through option `-Wd"-l [fn].ml" ' will also produce an fpp summary listing in `[fn].ml' (but no executable) with the markers `P` for autotasked, `V' for vectorized, `N' for not chosen or not optimized, adn `D' for data dependent.
f90 -O full -M [fn].f > [fn].m & (CR) : The `-M' option results in the intermediate Fortran file `[fn].m' with microtasking directives automatically inserted into the `[fn].f' source using the dependence analysis of the `fpp' preprocessor; no object or executable file is produced; the user can insert additional compiler directives into `[fn].m' and compile it with the Cray Fortran multitasking processor `fmp', the translator of the directives, by `fmp [fn].m > [fn].j (CR)'; the intermediate expanded file `[fn].j' is further assembled, linked and loaded by `sld -o [exec] [fn].j (CR)'.

• General MPI Information: Introductory Selections for MCS572
• MPI Web Man Pages.
• MCS572 MPI_Reduce Help Page.
• ANL Tutorial material on MPI available on the Web, by Bill Gropp et al at ANL.
• Getting Started With MPI: A Message Passing Interface for Parallel Programming.
• Using SHMEM on the CRAY T3E: Cray Native Shared Memory Message Passing Library.

• MCS572 PVM Example Page
• General PVM Information: Introductory Selections for MCS572
• PVM (Parallel Virtual Machine) Documentation at ORNL

[time-variable] = second() : The standard UNIX Fortran seconds timer utility, whose output value is user cpu time in seconds, as opposed to system ``cpu time'' and wall clock time (the sum of user and system times); also exists in the format `call second([time-variable])';for timing large loops, `second' overhead should be negligible; for most sizes of loops, the timing part of the code, with `!' marking comments on the statement line, might look like:

         real tv(100),cputim()                                         
         character*24 tchar(100)                                       
         kt = 1                                                        
         tv(kt) = second()      ! first 2 calls get the overhead       
         kt = kt + 1                                                   
         tv(kt) = second()      ! initial time                         
code-continues
          ...  more code ...
code-continues
         kt = kt + 1
         tchar(kt) = `loop [999]'
         tv(kt) = second()      
           do [999] i = 1, [1000]
code-continues
     ... rest of do .... 
code-continues
999      continue
         kt = kt + 1
         tv(kt) = second()      !tv(kt) - tv(kt-1) = do-cputime
code-continues
   ... more do loops and more timing step pairs .... 
code-continues
         kt = kt +1
         tv(kt) = second()            !final time
         overhd = tv(2) - tv(1)      !timer second overhead
           do [99999] ks =3, kt - 2      !cpu-time for each timed loop
         cputim(ks) = tv(ks+1) - tv(ks) -- overhd
         write(6,[99998]) ks, cputim(ks), tchar(ks)
Comment:  writes hinder vector optimization, so save writes until last
99999    continue
99998    format(1x,i3,' time =',f12.7,' for ',a)
         cputot = tv(kt) - tv(2) - (kt-2)*overhd
Caution: due to overhead variability, total can be off for small job
         write(6,*) 'total cpu-time =',cputot

For timing small loops, put the small loop inside another loop that just does a large number of repetitions of the small loop, say N, then divide the time difference by N; use `man second' for other information.
[flag] = gettimeofday(&tp,&tzp); : C/C++ microsecond wall clock timer and timezong utility. Requires the special header include statement: `#include ' and that the following structures be declared:

struct timeval tp ;
/* timeval is a structure with pointer name tp and having      */
/* unsigned long  tp.tv_sec giving  seconds since Jan. 1, 1970 */
/* long  tp.tv_usec giving microseconds                        */
struct timezone tzp;         /* needed only for time zone data */

See `man gettimeofday' for more information and Cray T90 C Starter example: `t90startcc.c'.}

[time-variable] = tsecnd() : f90 task timer utility giving the cpu time for a task during multitasking.

gettimeofday
Example C program using the gettimeofday function:

#include 
#include 
#define NTime 20

main()
{
/* Time variables */
   struct timeval tp ;
/* timeval is a structure with pointer name tp and having */
/* unsigned long  tp.tv_sec giving  seconds since Jan. 1, 1970 */
/* long  tp.tv_usec giving microseconds */
   struct timezone tzp; /* needed only for time zone data */
   int gtod;
   long int tsecs[NTime], tmicrosecs[NTime];
   long int ttot[NTime], ttotmoh[NTime];
   float ts1, tt1, tu1, tu2, ts2, tt2, tu3, ts3, tt3;
   double ttotf;

/* begin main code */
   if (gettimeofday(&tp,&tzp) == -1) { perror("gettimeofday failed"); exit(1);}
   kt = 1;
/* gettimeofday = Microsecond Wall Timer C function;                         */
/* WallTime = UserTime + SystemTime, Undecomposed;                           */
/* gettimeofday returns gtod = 0 if successful;                              */
/* tv_sec in secs since 1/1/70;                                              */
/* tv_usec in added microseconds;                                            */
/* tzp gives the timezone;                                                   */
   gtod=gettimeofday(&tp,&tzp);
   tsecs[0] = tp.tv_sec;
   tmicrosecs[0] = tp.tv_usec;
   ++kt;
   gtod=gettimeofday(&tp,&tzp);
   tsecs[1] = tp.tv_sec;
   tmicrosecs[1] = tp.tv_usec;
/*   ...... MUCH DELETED CODE ........ */

/*   ...... MUCH DELETED CODE ........ */
/* Clock: Elapsed Total Time: */
   ++kt;
   gtod=gettimeofday(&tp,&tzp);
   tsecs[kt] = tp.tv_sec;
   tmicrosecs[kt] = tp.tv_usec;
/* Total Elapsed Time Including Clock Overhead*/
   ttot[kt] = (tsecs[kt]-tsecs[1])*1000000+(tmicrosecs[kt]-tmicrosecs[1]);
/* Total Elapsed Time Minus Clock Overhead */
   ttotmoh[kt] = ttot[kt] - (tmicrosecs[1] - tmicrosecs[0]);
   printf("\nIntermediate Raw Timing Output:");
   printf("\ntmicrosecs[(0,1,kt)]=(%12d,%12d,%12d), in microseconds",
      tmicrosecs[0],tmicrosecs[1],tmicrosecs[kt]);
   printf("\ntsecs[(0,1,kt)]=(%12d,%12d,%12d), in seconds",
      tsecs[0],tsecs[1],tsecs[kt]);
   printf("\n(ttot[kt],ttotmoh[kt])=(%12d,%12d), in microseconds",
      ttot[kt],ttotmoh[kt]);
   if (ttot[kt] < 0){printf("\n  Error:Negative Times:Bad Clock:Rerun Job\n");}
   ttotf = ttotmoh[kt]/1.e6;
   printf("\n T90 Starter C Problem Output");
   printf("\n  Timing Output:");
   printf("\n   final total time=%12.4e, in seconds\n",ttotf);
/* Change:  Extra output statements: */
}

T90 (perhaps MPP) Timer Summary ... MCS572 F95/FBH
Timer     TimeMeasured      Units          Comments
-----     ------------      -----          --------
clock     System&User       Microseconds
cpused    User              ClockTicks     RTC ticks
gettimeofday    WallTime    Microseconds   plus many other things from TOD; C fn
ja        ElapsedUserSys    Seconds        plus more;on T90;only mppexec for T3E
rtclock   User              ClockTicks     current RTC ticks
RTC       RealTimeClock     ClockTicks     float version
IRTC      RealTimeClock     ClockTicks     int version
second    User              Seconds        Coarse, not useful for small timings 
secondr   ElapsedWall       Seconds        Coarse, not useful for small timings
sysclock  RealTimeClock     ClockTicks     plus #wraps (overflows)
timef     ElapsedWall       Milliseconds   Fn. gives elapsed time since 1st call
times     Process&Child     ClockTicks     needs include 
timex     ElapsedUserSys    Seconds        depends on opts in timex [opts] [cmd]
tsecnd    ElapsedTask       Seconds        for current multithreaded task

The Cray T3E at NPACI has a wall timer MPI_Wtime in seconds that works with MPI parallel programming codes for both f90 and cc codes. See the following information on MPI__Wtime and related functions:

• MPI_Wtime Man Page, Measurements in Seconds.

• MPI_Wtick Man Page, Resolution or Finest Time Interval Measured.

• Sample Fortran f90 Code Illustrating MPI_Wtime Usage, for Laplace-Jacobi Application.

• Sample C Code Illustrating MPI_Wtime Usage, for Laplace-Jacobi Application (under revision).

The best way to learn these commands is to use them in an actual computer session.