This User's Local Guide is intended to be a sufficient, hands-on introduction to the UIC Academic Computing and Communications Center (ACCC) HP9000/800 V2250 parallel processor called the "borg", the picture of this V-Class Exemplar Computer Server and its Star Trek namesake being found at the web link below:
The UIC HP9000/800 V 2250 has 14 parallel processors, collected into 7 pairs of 2 processors each on a chip. The local machine name is the `borg' and its full internet address is `borg.cc.uic.edu'. The HP9000/800 V2250 model denotes the Hewlett-Packard V-Class Exemplar Computer Server. The ACCC has the following brief information on the HP9000 borg:
particularly Figures 1-1 (Functional Block Diagram of a V-Class System) and 1-2 (ERAC Interconnection).)
The HP9000 system is configured in the attached figure:
The processor and memory are connected by an 8×8 Hyperplane CrossBar Switch using the 4 ERACs so that each processor is connected to all 32 memory banks. Each ERAC has a bandwidth of 1.9 GB/s (in GigaBytes per second), so that the overall bandwidth of the set of 4 ERACs is 15 GB/s. A diagram of the CrossBar switch is linked below:
Physical Memory: The physical memory or RAM (Random Access Memory) or global memory on the borg totals 4 GB (4 Giga Bytes or 4 × 1024&sub3; Bytes) of storage distributed over 32 Memory Banks with 64 MB (64 × 1024&sub2; Bytes). The partitioning of the Memory (RAM) into 32 memory banks is called "32-Way Interleaving" and allows for simultaneous memory access to different banks, which would not be possible if the entire memory was treated as one memory unit. Each array is stored over as many memory banks as possible. Each EMAC controls a set of 4 Memory Banks with a total of 256 MB of memory. The physical memory has a latency (delay) of 500 nanoseconds, approximately. The physical memory holds executing programs and data (called "Coherent Memory"), but also holds much system information. See also
particularly Figures 2-3 (Coherent Memory Space Layout) and 2-4 (Conceptual Layout of Physical Memory of a Fully Populated System).
Virtual Memory: Virtual memory is available through the use of memory units called pages, permitting the execution of much larger programs than can fit in physical memory by accessing by page segments of memory. The compiler generates the addresses of the much larger virtual memory, but only those pages currently executing are translated into physical memory. The HP PA-RISC 8200 translates both 32-bit and 64-bit virtual and physical memory addresses into 64-bit addresses. The Translation Lookaside Buffer (TLB) is the hardware device that helps translate or map virtual memory references to physical memory pages to permit valid memory accesses. See also:
Cache: In addition, each processor has a 2 MB of fast data cache memory, a 2 MB of fast instruction cache memory, making a total of 4 MB each. Cache memory is generally smaller, faster and closer to the CPU than physical memory. (See also
Hard Disk Memory: Hard disk memory consists of 8 × 9 GB Ultra SCSI disks, making 72 GB total. On these disks your user directories, operating system, other system software, and application software are stored. Hard disk memory is generally bigger, slower and furthest away from the CPU than hard disk memory. Hence, the memory hierarchy is the order from cache to physical to hard disk memory. Users can use up to 50 MB of hard disk memory in their home directory (`/homes/home7/[userid]'). The 50 MB is a soft quota and 75 MB is the hard quota, presumably for temporary storage. Users also have a scratch directory in `/scratch/[userid]' of up to 500 MB, but scratch memory is temporary and volatile subject to systems administration erasure. Users can check out their quotas, by the command:
quota -v
{Yet no HP9000 is powerful enough to appear on the current Top500 Computer Report (report is big)
Note that units of the Hockney asymptotic performance rates, actual maximum Rmax and vendor peak Rpeak, are in MegaFlops, while Nmax is the maximum solved matrix corresponding to Rmax and N1/2 is the matrix size corresponding to half Rmax.
Another benchmark using the NASA NAS BT Simulated Computational Fluid Dynamics Application does not list any HP9000. Gunter's benchmark site:
The HP9000 would be classified as a mini-supercomputer, not quite a supercomputer which denotes the class of the most powerful machines. In fact,
only mentions Supercomputing Technologies are used.
However, it offers local access and many of the parallel constructs found on massively parallel machines.
and very large (over 200 pages or 13 MB)) official guide from Platform Computing Corporation accessible by ftp (file transfer protocol: see below) to the borg using your borg account at the location:
/usr/local/lsf/doc/lsf3.1_batch.user.ps
However, this user guide and the brief UIC LSF description should be sufficient to start with.
The basic execution envelope command is lsrun, the UIC Systems Administration requests that you run your computer jobs with lsrun, and it is found in the directory path:
/usr/local/lsf/bin/lsrun
which is a virtual link to the physical command path:
/usr/local/lsf/mnt/hppa/bin/lsrun
as are most LSF commands. A typical use would be of the form:
lsrun [executable] >& [output_file]& ,
where "[executable]" is your executable file run in the background (>&) and "[output_file]" your output file where output is redirected (&).
Other LSF commands are given below in the section:
TELNET: Access to the `borg' is by either the universal TCP/IP Internet Protocol:
rlogin borg.cc.uic.edu -l [borg_user_name]
ssh borg.cc.uic.edu -l [borg_user_name]
FTP: File transfer to the `borg' is best accomplished by the universal TCP/IP Internet Protocol command:
Caution: If CMS is your home computer system, then you are restricted to file transfers only from CMS to the new `borg' while you are logged onto CMS while accessing the new `borg'. This is because CMS accounts are single user accounts restricted to single access only. That is, FTP to and Telnet from CMS would count as a double access so is not permitted in CMS. This is very different from Unix which is a multiuser system allowing multiple access, i.e., you can even log into the new `borg' several times simultaneous, but you also degrade your own performance.
/opt/fortran90/bin/f90
and the HP-UX optimizing C compiler cc, found the new `borg' directory location:
/opt/ansic/bin/cc
The F90 options
FORTRAN: A typical format for compiling and linking a HP Fortran program named `[program].f' with parallel optimizations {`+O3'} in the background {last `&'}, provided the PATH variable is set properly to the system value, is
lsrun f90 +O3 +U77 +list -o [executable-file] [program].f >& [program].LIST&
where `lsrun' is the `borg' Load Share run command. See also:
.where both compiler information listings {`+list'} and error messages {`>&'} are redirected into the file `[program].LIST', timer utilities are permitted {`+U77'}, while the executable module is renamed `[executable-file]'. It is assumed that the user replaces the square brackets and their contents by actual legal names.
On the `borg' enter
In summary, the optimizing Fortran-Compilers are
Command Version Full Path/Location -------- ------------------- ---------------------------- f90 HP-UX Fortran 90 /opt/fortran90/bin/f90 (use) f77 HP-UX Fortran 77 /opt/fortran/bin/f77
C: Similar format is used to compile and link C code, provided the PATH variable is properly set to the system value:
lsrun cc +O3 +L -lm -o [executable-file] [program].c >& [program].LIST&
where `lsrun' is the `borg' Load Share run command,
and where compiler information listings and
error messages {`>&'} are redirected into the file
`[program].LIST' enabled by `+L' option, while the executable module is renamed
`[executable-file]' and `-lm' allows access to the function of math library
{`#include
C-Version Caution:
There are multiple versions of the C-compiler.
The `cc' HP C Compiler is the one in `/opt/ansic/bin/cc',
and, in fact, the HP-UX Standard C
Compiler is the one in `/bin/cc' is actually link to the optimizing
one, which is what you want to happen.
There also the GNU (Glad its Not Unix public domain) Project version
`gcc' in
`/opt/langtools/dde/softbench/newconfig/opt/softbench/config/compiler_info/gcc'
is not yet enabled.
In summary, the
C-Compilers are
C++:
Note: both c++ and g++ do not seem to be currently implemented, but
the optimizing ANSI C++ (aCC) is available. The `aCC' takes some of
the same options as `cc'.
See also:
Shell Caution: If you have trouble with the above compiler
command syntax, it may be that you have your borg account shell
set as the Bourne shell (ksh) rather than the user standard C-shell
(csh). You can change you shell for the current session by the
command
/bin/csh
or you can change the shell permanently to a C-shell for which this guide is
written by the command:
chsh [borg_user_name] /bin/csh
where `[borg_user_name]' must be replaced by your actual borg
account name, but you must also logout and logback in again to
activate the shell change. Using a C-shell should make your access
must easier, while the Korn shell (ksh) or generic shell (sh) would be
more powerful if you were doing computer systems work rather than
scientific computing. New student accounts are set up with a Korn
shell (ksh) in the ACCC template, probably because that is what the
systems people use. See the
SHELL PATH section below or consult the
ACCC page:
XTERM: The default terminal emulation is MIT's X-Windows
systems which is typically used with access from Sun Unix or IBM
AIX workstations permitting the windowing interface of these
workstations to pass through to the remote Exemplar.
TERM vt100: However, if you are not using X-Windows to
access the `borg', then it is suggested that you use the
DEC `vt100' terminal emulation by entering the Unix command
VI: If your are also going to be using the usual Unix
visual editor `vi', see the following manual (man) pages:
THE: There is also The Hesseling Editor `the',
which is like the CMS xedit command, so use the manual command:
PICO: In addition, there is also the `pico'
editor from the `pine' mail program, but `pine'
itself is not on the `borg' since you are permitted only
to send mail from the `borg' but not to it.
To use `pico', it is suggested that you use the simple display
system `pilot', mentioned below, that uses `pico' to
implement its `Edit' submenu command .
There are man (manual) pages accessible by the commands:
man pilot
man pico
setenv PATH "$PATH"
setenv MANPATH "$MANPATH"
set path=( $path )
set MANPATH=( $MANPATH )
These forms are so that you inherit the system file path $PATH and manual
path $MANPATH as initializations for the C-SHELL (form is different if you
use the K or Bourne-Shells), and make sure that you have the paths for
the optimizing f90, cc, cpp (C Preprocessor) and lsrun commands.
You may even append custom paths to the system paths.
lsrun: On the `borg' at UIC, a custom environment
so that you do not have to do it with each user
command called `lsrun' greatly simplifies the execution of programs
(the full path is `/usr/local/lsf/bin/lsrun'.
The Computer Center requires use of `lsrun'.
A typical format for executing a module named
`[executable-file]' in the background {last `&'} is
Users can also use lsrun for the compile/load step as well.
pilot
which is fairly self explanatory using the `Arrow Keys' to move around
the file display and using commands like
Launch:
The "Launch (l)" command uses the Bourne Shell (sh) instead of the simple
user C-Shell (csh), so the above Compile/Link (f90 or cc) or Load Share
Run Executable (lsrun) commands must be modified slightly for `sh' syntax
that differs from `csh' syntax. Basically, you need to
replace "[command] >& [output]" by "[command] > [output] 2>&1"
Here, the "2>&1" means merge standard error (2=stderr) with standard output
(1=stdout, here redirected to the file "[output]"), not as simple as
the csh form but more precise, which is why systems people prefer sh or
other shells and users usually prefer the simpler csh. }Note: Standard
input (0=stdin) can also be used.}
On the borg type:
{So for a program called for example pgm.f or pgm.c, then
Compile/Load, type after "Command to execute:"} lsrun f90 +O3 +U77 +list -o pgm pgm.f > pgm.LIST 2>&1
lsrun cc +O3 -lm +L -o pgm pgm.c > pgm.LIST 2>&1
{Run Executable, type after "Command to execute:":} lsrun f90 +O3 +U77 +list -o pgm pgm.f > pgm.LIST 2>&1 when done, the file display will not automatically update and there
was no pilot refresh command, but using the `arrow keys' to put the
cursor on the parent directory `..' or any directory and typing `vv' for
double view does work as a crude file display updator,
move cursor to `pgm.LIST' by `alrow keys', type `e' for edit to see compile
listing contents,
lsrun pgm > pgm.output 2>&1 Edit: The Edit command within Pilot (really Pico)
uses EMACS like ctrl commands like `^Key' which means the `Ctrl-Key'
pressing together (or at least `Ctrl' held then
LOWER CASE `Key' pressed:
^G => Display help. {I.e., use `Ctrl-g'.}
^L => Refresh the display.{I.e., use `Ctrl-l'. You
will need this often to see your editing}.
^X => Exit pico, saving buffer. {I.e., use `Ctrl-x'.}
^W => Search for (where is) text, neglecting case.
{I.e., use `Ctrl-w'.}
Arrow Keys => Move cursor up, down, forward or backward
using the appropriate `Arrow Keys', then press `^L' = `Ctrl-l' to make
sure what you have done.
^F => Move Forward a character. {I.e., use `Ctrl-f'.}
^P => Move to the Previous line. {I.e., use `Ctrl-p'.}
^A => Move to the beginning of the current line.
{I.e., use `Ctrl-a'.}
^V => Move forward a page of text. {I.e., use `Ctrl-v'.}
type => There is no insert command, so just move the cursor
to the place you want by the `Arrow Keys' or other Move Ctrl-Keys and just
type in the characters that you want inserted.
^D => Delete the character at the cursor position.
{I.e., use `Ctrl-d'.}
^^ => Mark cursor position as beginning of selected text.
{I.e., use `Ctrl-^'. Note: Setting Mark (i.e., pressing `^^' again)
when already set unselects text.}
^K => Cut Marked text (displayed in inverse characters).
{I.e., use `Ctrl-k'. Note: The Marked text's boundary using mark
`^^' command on the cursor side
ends at the left edge of the cursor. So, with
Marked text to the left of the cursor, the
character under the cursor is not selected. The complete procedure is
of the form `^^, ^F, ..., ^F, ^K' with n `^F's for cutting out
n characters.}
^I => Insert a tab at the current cursor position.
{I.e., use `Ctrl-i'.}
^J => Format (justify) the current paragraph.
{I.e., use `Ctrl-j'.
Note: paragraphs delimited by blank lines or indentation.}
^T => To invoke the spelling checker.
{I.e., use `Ctrl-t'.}
^C => Report Current Cursor position.
{I.e., use `Ctrl-c'.}
^R => Read in or insert an external file at the current cursor position.
{I.e., use `Ctrl-r'.}
^O => Output the current buffer to a file, saving it.
{I.e., use `Ctrl-o'.}
Shell scripts for borg Command Line or pilot Launch (l) command have been
place in the borg directory `/homes/home7/hanson/' that can be used to
compile and execute programs without concern for which shell is being used,
since the shell selection is forced in the script.
Here is a directory listing of the files:
cp ~hanson/*sc .
{"*sc" is UNIX short hand for all the script files and the "." denotes your
current directory where you are assumed to want the files.}
chmod u+x *sc
cp [source].f pgm.f OR cp [source].c pgm.c
rm pgm pgm.o pgm.LIST pgm.output
{The compile/load scripts also do this, but it is there
for you to just clean up your files.}
pilot {call for the new version of pilot}
l {to launch and executable or script}
"Command to execute:" F90.sc
{optionally, you can move the arrows to `pgm.LIST' and use
the pilot Edit (type `e') command to check if there are not
compile/link errors, using `Ctrl-x' to quit Edit.}
"Command to execute:" RUN.sc
{optionally, you can move the arrows to `pgm.output' and use the Edit
(type `e') pilot command to check your output,
using `Ctrl-x' to quit Edit.}
q {to quit pilot}
A personal account on the `new borg' is available to any
faculty member who requests one and to any student who has obtained
a faculty member's sponsorship permission. Note that the UIC HP9000 is not
intended to be used for general needs such as email, news reading,
or Web browsing; its resources should be used only for computationally
intensive needs. If you wish to obtain an account please send email
to
Use those optimizing compilers found in the `/opt' directory.
Command Version Full Path/Location
------- --------- -----------------------
cc HP-UX C /opt/ansic/bin/cc (use)
cc CCS C /usr/ccs/bin/cc (not enabled)
cc Std. C /usr/bin/cc -> /opt/ansic/bin/cc
cpp HP-UX cpp /opt/langtools/lbin/cpp (C Preprocessor)
aCC HP-UX aC++ /opt/aCC/bin/aCC (use)
c++ HP-UX c++ /usr/local/bin/c++ -> /usr/local/src/gcc/bin/c++
gcc GNU C Not executable
g++ GNU C++ /usr/local/bin/g++ -> /usr/local/src/gcc/bin/g++
Editors and Emulations.
setenv TERM vt100
on the `borg' command line, or better replace your `set
xterm' line in your Login resource configuration files `.login' for C-Shell
(or `.profile' for K-Shell) on the `borg'.
man vi
along with the vt100 emulation then you will want to replace your
Edit Resource Configuration file `.exrc', which by default template
is for the `emacs' {"Ensures Maximally, Almost, Carpal-tunnel
Syndrome" {%>)}; an editor favored by computer systems people}, by
the following version:
In fact, since certain execution commands force the vt100 emulation
to be revert back to `xterm', a better fix would be to redefine
the `vi' editor command in your C-SHell Resource Configuration file
`.cshrc', by trying the command :
vi .cshrc
by adding the following line among the aliases:
alias vi 'setenv TERM vt100; vi'
Using the write and quit command:
:wq
within vi and activating the change by entering on the `borg'
command line the source execution command:
source .cshrc
Then the editing command:
vi [your_file]
should work as usual.
man the
for more information.
Manual and File Paths.
Users can also use the forms suggested on the
ACCC hpborg.html:
Load Share Program Execution.
lsrun [executable-file] >& [output-file]
&
where both calculated output and error messages {`>&'} are
redirected into the file `[output-file]'.
Pilot: Simple File Display & Execute Browser.
The browser displays a file list at the top of the screen with a the
bottom few lines indicating the available commands just listed above.
{Some more Hints:
^B => Move Backward a character. {I.e., use `Ctrl-b'.}
^N => Move to the Next line. {I.e., use `Ctrl-n'.}
^E => Move to the End of the current line. {I.e., use `Ctrl-e'.}
^Y => Move backward a page of text. {I.e., use `Ctrl-y'.}
^U => Uncut (paste) last cut text inserting it at the
current cursor position, or use as an Undo by using at the same position.
{I.e., use `Ctrl-u'.}
Shell Scripts for Runing Programs.
borg 91: ll *sc
-rwxr-xr-x 1 hanson math 536 Oct 17 00:00 CC.ksc
-rwxr-xr-x 1 hanson math 510 Oct 17 00:03 CC.sc
-rwxr-xr-x 1 hanson math 264 Oct 17 00:12 CLEAN.ksc
-rwxr-xr-x 1 hanson math 260 Oct 17 00:15 CLEAN.sc
-rwxr-xr-x 1 hanson math 542 Oct 17 00:03 F90.ksc
-rwxr-xr-x 1 hanson math 512 Oct 17 00:04 F90.sc
-rwxr-xr-x 1 hanson math 319 Oct 17 00:00 RUN.ksc
-rwxr-xr-x 1 hanson math 304 Oct 17 00:16 RUN.sc
They execute on the `borg' command line or with the `pilot' Launch (l) command.
ACCC consultant said new student accounts are set up with the K-Shell ksh.
Here is a procedure for using these scripts:
foreground: background:
F90.sc F90.sc &
RUN.sc RUN.sc &
{Caution: your have to wait for the compile to finish, before the run.}
UIC borg HP9000 Accounts.
borg account request.
Include your netid, your telephone number, and a brief description
of your intended usage. Students must also provide the name and
email address of the faculty member who is sponsoring the account,
i.e., students should send in the message the following items:
(For MCS 572 students, the Intended Usage is MCS 572 Introduction to Supercomputing, with Faculty Sponsor Prof. F. Hanson)
This mini-local-guide is meant to indicate ``what works'' and what is ``useful'' for beginning users. The guide also gives alternate methods for access from UNIX systems.
HP and UX are trademarks of Hewlett-Packard Company. UNIX is a trademark of X/Open Company, Inc.
This guide is intended t be self contained, but users who want further information, can consult the following sources (you can just click on the highlighted topics to access if you are surfing the world Wide Web):
For example, search on `Gwennap' to get an HTML reprint of the paper by Linley Gwennap, yields:
The login procedure depends on your local method of accessing the HP9000 from UIC, but the best access is from a Unix type system since the UIC HP9000 operating system is HP-UX, which is substantially Unix and it is to the user's advantage to use Unix to Unix communication. If you do not now have a Unix account you should try to get one from your department's Unix system or from the UIC Computer Center graduate student Unix (Sun Solaris Unix) server called `icarus'. Unix workstations are available in many science and engineering departments. Communication using the IBM mainframe suffers from horrible terminal emulation problems, so that you should avoid it if you can. If that does not work out or is not practical see Professor Hanson about other alternatives.
UNIX Access: The ACCC preferred remote telnet command:
telnet borg.uic.edu
rlogin borg.uic.edu -l [borg_user_name]
and is best since it passes your local terminal emulation better from the local session through to the remote UIC one and you do not have to enter your login name as with `telnet' and can proceed directly to the `password' step below.
PCLAB Access: Access by `telnet' TCP/IP command also works for PCLab PCs, or CMS, as well as in Unix, with the format:
telnet borg.uic.edu
{Caution: From a UNIX operating system, it is essential to use lower case; borg.cc.uic.edu is the full Internet name for the UIC HP9000, but the shortened name borg.cc.uic.edu also works; the corresponding Internet Number of the new `borg' is `128.248.155.55' (the old borg is `128.248.100.55') and is more basic since the Internet Name is derived from the number and the number may work when the UIC computer domain name server (DNS) is down; the new `borg' should respond with:}
``login:'' [borg_user_name] (CR) {Enter your actual new `borg' user name for `[borg_user_name]' on the HP9000, the name supplied by the computer center when you apply for the MCS572 class or for research with your faculty advisor; whether `telnet' or rlogin; Next}
``password:'' [password] (CR) {Enter your actual user password in place of `[password]' after the password prompt, again in lower case at the cursor. If you make a mistake or are telnetting from an account with a different user name, you will probably have to try both login-id and password again.}
``...system Message Of The Day (MOTD) and Copyright information...''
``borg[n]: '' {The `borg[n]: ' is the default HP9000 `borg' prompt and `[n]' is some command line counter which we will ignore in the sequel. You are now in HP-UX on the UIC HP9000. The UIC Computer Center, as are almost all computer centers, is very serious about computer security, so the first thing you should do is to change the UIC temporary password originally given to you by entering the password change command}
passwd (CR)
``Old password:'' {Enter your original password again.}
[old-password] (CR)
``New password:'' {Enter your new 8 character password, which should contain at least two alphabetic characters and at least one numeric or special character.}
[new-password] (CR)
``Re-enter new password:'' {Retype new password to confirm original typed change.}
[new-password] (CR)
``borg: '' {Congratulations, you made it to the UIC HP9000; have a nice session.
LOGGING OUT:
You can end this session at any time you have a `borg: ' prompt
by entering:
logout (CR)
or pressing the `Ctrl' control key and
'd' key simultaneously (i.e., `ctrl-d', but you can hold down the
`Ctrl-key' and then press the `d-key').
FILE COMMANDS: You can check what the name of your `borg' home directory (file system) is by the Unix ``print working directory'' command:}
``borg: '' pwd (CR) {Your disk directory should be something like `/homes/home7/[borg_user_name]' where `[borg_user_name]' is your HP9000 Login-id on the UIC HP9000. You can list the current files on your account by the Unix ``list sets'' command:}
``borg: '' ls (CR) {If this is a new account you probably will not have any regular files listed, but the following form of `ls' command has options that reveal hidden `dot' files and give the long form of file information:}
``borg: '' ls -al (CR) {You may continue with a HP-UX session by getting help with the usual UNIX `man' manual command:}
``borg: '' man ls (CR) {The default `man' output is paged, so press the `Spacebar' or enter `d' for another page and enter `q' for quit. Try `man man' for more information.}
VI SETUP: If you are going to be using the Unix `vi' visual editor rather than the `emacs' editor or an X-Windows editor via `xterm', then you might want to set your terminal environment to the standard `vt100' terminal emulation by the command:
``borg: '' setenv TERM vt100 (CR)
Along with this change to `vt100' (or `vt132'), you should first save your old Unix edit (`EX') resource configuration file `.exrc' using the Unix copy `cp' command :
``borg: '' cp .exrc .exrc_old (CR)
to a `vt100' friendly one (if permissions permit from Professor Hanson's account) by
``borg: '' cp ~hanson/.exrc_vt100 .exrc (CR)
or by using the web link with a web browser at your accessing computer to get a copy:
although you might have to use the File Transfer Protocol `ftp' to get it to the `borg', for example by Anonymous FTP:
``borg: '' ftp www.math.uic.edu (CR)
``Name (www.math.uic.edu:[user]): '' anonymous (CR)
``Password: '' [send_email_identity_as_password] (CR)
``ftp> '' cd pub/Hanson/MCS572 (CR)
``ftp> '' get .exrc_borg .exrc (CR)
``ftp> '' quit (CR)
FTP will be described below in Section on FTP.
FORTRAN SESSION: For a sample session for compiling and executing a Fortran Program, you can get a copy of the MCS572 `borg' starter problem via the web and transfer it to the borg:
or by `cp' copy command:
``borg: '' cp ~hanson/borgstart.f start.f (CR)
or by Anonymous FTP:
``borg: '' ftp www.math.uic.edu (CR)
``Name (www.math.uic.edu:[user]): '' anonymous (CR)
``Password: '' [send_email_identity_as_password] (CR)
``ftp> '' cd pub/Hanson/MCS572 (CR)
``ftp> '' get borgstart.f start.f (CR)
``ftp> '' quit (CR)
{Note: if this Anonymous FTP method is used than the getting of `.exrc' above and `start.f' here can be combined.}
For an example of compiling and linking the MCS572 Fortran based `borg' starter problem (assuming `start.f' has been transferred to your `borg' account), enter:
``borg: '' f90 +O3 +U77 +list -o start start.f >& start.LIST& (CR) {Here,
``borg: '' cc +O3 +L -lm -o start start.c >& start.LIST& (CR) {Note that the optimization option is only `+O3' and that there is no report option. The status of the job, which ever programming language, can be determined form the Unix `jobs' command:}
``borg: '' jobs (CR) {When you get the message:
``Done f90 +O3 +U77 +list -o start start.f >& start.LIST''
instead of ``+Running ...'', then the listing file can be examined by the Unix `more' paging command:}
``borg: '' more start.LIST (CR) {Else, the listing file can be viewed by the Unix visual editor (or other favorite editor):}
``borg: '' vi start.LIST (CR) {If compilation and listing is satisfactory, then the module `start' may be executed using the UIC custom environment `lsrun' command preceding the `start executable rather than directly using `start':}
``borg: '' lsrun start >& start.output & (CR) {If the proper directory `/usr/local/lsf/bin' for `lsrun' is not in your defined directory path, then you can add it to your C-Shell Resource Configuration file `.cshrc' or you can use the full name of lsrun:}
``borg: '' /usr/local/lsf/bin/lsrun start >& start.output& (CR) {Again, the job status may be checked while waiting by entering:}
``borg: '' jobs (CR) {When the ``Done'' message is displayed rather than ``+Running ...'', then the output can be viewed in pages:}
``borg: '' more start.output (CR) {Caution: the executable `start', as with the default executable `a.out', is a binary, rather than text file, so is not readable. Next you can transfer your files back to your printer connected computer (Computer Center would be prefer that you not print from the `borg') by FTP:}
``borg: '' ftp [home_machine].[department].uic.edu (CR)
``login: '' [user_name] (CR)
``password: '' [user_password] (CR)
``ftp> '' cd [target_directory] (CR)
``ftp> '' put start.f (CR)
``ftp> '' put start.LIST (CR)
``ftp> '' put start.output (CR)
``ftp> '' quit (CR) {Finally for the job, it is good file management to remove your executable file since it is cheaper to regenerate than to store, so as not to be a "storage hog" using the Unix remove `rm' files command:}
``borg: '' rm start (CR) {At this point you can `logout' of the `borg' by entering:}
``borg: '' logout (CR)
``% '' {Return to your local UIC UNIX session.}
C SESSION:
You can run a sample HP C program by transferring to the `borg' a starter C code version:
or by `cp' copy command:
``borg: '' cp ~hanson/borgstartcc.c start.c (CR)
or by Anonymous FTP:
``borg: '' ftp www.math.uic.edu (CR)
``Name (www.math.uic.edu:[user]): '' anonymous (CR)
``Password: '' [send_email_identity_as_password] (CR)
``ftp> '' cd pub/Hanson/MCS572 (CR)
``ftp> '' get borgstartcc.c start.c (CR)
``ftp> '' quit (CR)
{In order to compile and link this C code, enter:}
``borg': '' cc +O3 +L -lm -o start start.c >& start.LIST & (CR) {Here, `cc' is the optimizing HP-UX optimizing C compiler (`/opt/ansic/bin/cc'), `+O3' is the parallel optimization option (Caution: the optimizing compiler must be the one in `/opt/ansic/bin', so to make sure execute the command `which cc (CR)' and `/opt/ansic/bin' be displayed.), with output along with error messages going to the file `start.LIST' according to the redirection option `>&', `start.c' is the C source file, `-o start' means that the execution file is named just `start', and the last `&' means compilation is done in the background permitting use of the terminal while waiting for the job to finish. The status of the job can be determined form the Unix `jobs' command:}
``borg: '' jobs (CR) {When you get the message:
``Done cc +O3 +L -lm -o start start.c >& start.LIST''
instead of ``+Running ...'', then the listing file can be examined by the Unix `more' paging command:}
``borg: '' more start.LIST (CR) {If compilation and listing is satisfactory, then the module `start' may be executed using the UIC custom environment `lsrun' command preceding the `start' executable rather than directly using `start':}
``borg: '' lsrun start >& start.output & (CR) {If the proper directory `/usr/local/bin' for `lsrun' is not in your defined directory path, then you can add it to your C-Shell Resource Configuration file `.cshrc' or you can use the full name of lsrun:}
``borg: '' /usr/local/lsf/bin/lsrun start >& start.output & (CR) {Again, the job status may be checked while waiting by entering:}
``borg: '' jobs (CR) {When the ``Done'' message is displayed rather than ``+Running ...'', then the output can be viewed in pages:}
``borg: '' more start.output (CR) {Next you can transfer your files back to your printer connected computer (Computer Center would be prefer that you not print from the `borg') by FTP as about with the `start' sample. Remember to remove your large executable:}
``borg: '' rm start (CR) {At this point you can `logout' of the `borg' by entering:}
``borg: '' logout (CR)
``% '' {Return to your local UIC UNIX session.}
The FTP file transfer protocol is the fastest method of file transfer between UIC and UIC HP-UX, because it uses a fast internet communication link.
{Caution: FTP involving UIC CMS should be initiated from CMS, because you can not have multiple write links to the same CMS disk (multiple read links should be OK. There are no similar problems using UNIX system, because UNIX is a multi-user system.}
At the UIC HP9000 you can transfer file between the HP9000 and UICVM or UNIX, even the NPACI/SDSC Cray T90, NPACI/SDSC Cray T3E or NCSA Cray Origin if you have an account there. The `ftp' command on UNIX is very much like the `ftp' command in HP-UX.
In order to transfer a file from HP-UX and to UIC, enter the commands:
``borg: '' ftp [machine].[dept].uic.edu (CR) {where you must enter the actual machine-department "[machine].[dept]" name; {Caution: some machines will have more than one part to the name}}
``Name ([machine].[dept].uic.edu:[UIC-name]): '' [UIC-name] (CR)
``Password: '' [UIC-password] (CR) {If you make a mistake at this point and you get an `ftp> ' prompt then you can either enter `quit' and start again or enter `user [UIC-name]' continuing:}
``ftp> '' cd [UIC-directory] (CR) {change directory to the target UIC directory.}
``ftp> '' lcd [HP-UX-directory] (CR) {locally change directory to the target UIC HP-UX directory, but note that the meta variables `$HOME' or `$TMP' will not work here and you will need the full path name since FTP is an entirely different session! Hence it is best to initiate FTP from the HP-UX directory from or to which the transfer will be made. For the home directory the the abbreviations `~' or `~[name]' will still work even if `$HOME' does not.}
``ftp> '' ls (CR) {Lists files in the UIC directory; wild-cards can be used.}
``ftp> '' !ls (CR) {Lists files in the local HP-UX directory using the `!' escape to the local HP-UX session; wild-cards can be used.}
``ftp> '' !get [UIC-fn.ext] [[HP-UX-fn.ext]] (CR) {Gets file from the remote UIC directory and save in the local HP-UX directory with optional new name `[HP-UX-fn.ext]'; wild-cards cannot be used, but the ftp multiple get command `mget' does permit wild-cards.}
``ftp >'' put [HP-UX-fn.ext] [[UNIX-fn.ext]] (CR) {Puts file from the local HP-UX directory and save in the remote UIC directory with optional new name [UNIX-fn.ext] ; wild-cards cannot be used, but the ftp multiple get command `mput' does permit wild-cards.}
``ftp >'' quit (CR) {Quit when done transferring files.}
``borg: ''
``borg: '' ftp uicvm.cc.uic.edu (CR)
or
``borg: '' ftp 128.248.2.50 (CR)
{This command allows you to enter the FTP communication system that uses the same lines and protocol as Telnet, but essentially only allows file transfer. The Internet numbers are more reliable. In the HP-UX to UICVM FTP connection, you will be prompted for your CMS user-id and your CMS password:}
``Connected to uicvm.uic.edu'' {If you do not get connected but end up in FTP you can try 'open 128.248.2.50' without restarting in HP-UX again.}
``Name(uicvm.cc.uic.edu:u[default-id]):'' [CMS-user-id like `[userid]'] (CR)
``Password(uicvm.cc.uic.edu:[CMS-user-id]):'' [CMS password] (CR) {If successful, then:} {If you make a mistake with either your password or username, you can enter `user (CR)' after the ``ftp>'' prompt to restart. At the HP-UX FTP ``ftp>'' prompt (it differs from the IBM FTP prompt), you can issue FTP commands:}
``ftp >'' help [FTP-command] (CR) {This `help' command gives a short information or definition of the command `[FTP-command]'; `help', alone, will display a list of FTP commands; `?' is an brief alias for `help'.}
``ftp >'' ls (CR) {Either `ls' or `dir' list the current contents of the remote directory if you need more information. `pwd' displays the remote (HP-UX here) working directory.}
``ftp>'' ls *.fortran (CR) {This example causes the listing of Fortran files on your CMS disk, with the wild-card `*' standing for any filename. Similarly, use `ls *.f (CR)' in UNIX.}
``ftp >'' put [HP-UX-fn.ext] [UICVM-fn.ft.fm] (CR)
or {The put command stores the local (HP-UX) file on the remote (UICVM or UNIX) system in an FTP session started from HP-UX. `send' is an alias for `put', while `mput [HP-UX-files] (CR) is used to send multiple files to UIC with the similar names.}
``ftp >'' get [UICVM-fn.ft.fm] [HP-UX-fn.ext] (CR) {The GET command transfers a file from UIC to HP-UX If successful, you should get messages like this:} {`recv' is an alias of `get'.and `mget [UIC-files] (CR)' is used to receive multiple files quickly with wild-cards, but the file names will be the same as they are on the remote machine. Caution: `mget' stores the CMS file into HP-UX with an upper case name, unlike `get' which stores into HP-UX properly with a lower case name. You can transfer some more files if you want, changing the directory if needed, or you can quit FTP by using the command:}
``ftp >'' quit (CR) {You can also use `bye' to exit FTP, except in IBM FTP. Either will get you back to the HP-UX (UNIX+) shell with prompt ``u* *%''.}
``borg: ''
The file transfer protocol program from a UNIX session is a similar to file transfers from the UIC HP-UX sessions, because both have UNIX or extended UNIX operating systems, as discussed in the last section.
File transfer protocol (ftp) on a PC Lab PC may not be practical for must users, due to lack of permanent storage. Transfer between CMS or UNIX and the HP9000 may be more practical when you are accessing them using `telnet' from the PCs. The nearest Xerox PostScript printer to 2249f is SEL2263, while others are SEL2265, SEL2058, SEO308 and elsewhere. However, if the PC is your favorite medium, then use it as in the above HP9000 or Unix subsections.
At UIC the IBMNET version of FTP is used on CMS and it uses the following commands:
GETDISK TELNET (CR) {This is the first step. It accesses the IBM TELNET disk that contains both FTP and IBM TELNET commands, so if you already have accessed the IBM TELNET disk for TELNET in your UICVM session, you do not have to get it again for FTP, and vice versa. For more information on FTP, type `HELP FTP' or `HELP FTP MENU'.}
FTP borg.cc.uic.edu (CR)
{This is the usual 2nd step, unless you have already accessed the Telnet disk in your session. This brings up the file transfer protocol system so that you can transfer files and connects to the UIC HP9000. If you do not get the HP9000 Internet name or number correct and do not get connected, you can still open a connection to the HP9000 using the command:
OPEN borg.cc.uic.edu (CR) You still have to log into the HP9000. You can get brief help in FTP by typing `?' after the `Command:' FTP prompt. However, next you get the FTP banner and login requests.}
``VM TCP/IP FTP R1.1''
``Connecting to BORG-FE.UIC.EDU 128.248.155.55, port ...''
``220 FTP server ( ... ) ready.''
``USER (identify yourself to the host):'' {The user must enter his HP9000 user-id of the form `u[?]' or `[userid]' assigned on the UIC ``Sign-on'' sheets.}
[HP9000-user-id] (CR)
``>>>USER u[?]''
``331 Password required for u[?]''
``Password:'' {FTP and HP9000 request the account password.}
[password] (CR)
``>>>PASS ********''
``230 User u[?] logged in.''
``Command:'' {This is the IBM Telnet FTP prompt, so at this point you can enter such commands as help, get, put, cd, ls, open, binary or quit.}
``Command:'' help [FTP-command] (CR) {The HELP command without any arguments displays a list of FTP commands; otherwise, with an argument, gives the syntax for `[FTP-command]'. `?' also gives short useful descriptions of FTP subcommands.}
``Command:'' get [HP-UX-fn.ext] [UICVM-fn.ft.fm] [(REPLACE] (CR) {The GET command gets the HP9000 file [HP-UX-fn.ext] and stores in into the CMS file [UICVM-fn.ft.fm]. E.g., `get pgm.l HP9000_pgm.listing' transfers the HP-UX compiler program listing to your CMS disk and renames it `HP9000_pgm.listing'. Note that inside FTP you must use `.' to separate UICVM CMS name, type and mode, not like the format in CMS. HP-UX Unix files usually have `.' separating name and additional extensions. If the UICVM CMS file already exists, the `(REPLACE' option must be used to over-write it. There is a form `mget [Unix-files] [(REP] (CR)' for getting multiple files in a single command. Caution: the `get' command assumes the working or root directory, otherwise you must change the UNIX directory using the `cd' command. Several carriage control characters do not transfer very well from computer system to computer system, especially tabs {Warning: DO NOT use tabs for files, except for `makefiles', because tabs will only cause you problems upon file transfer.} In CMS XEDIT, you can remove trouble-some carriage control characters by typing `set hex on (CR)' on the bottom command line, and then changing tabs to a set of blanks {4 blanks is a good number} by typing `c/x'05'/ /* * (CR)', or deleting underscores by `c/x'6d16'/* * (CR)', or deleting just backspace characters by `c/x'16'/* * (CR)'. Warning: type only the quotes surround the Hex numbers.}
``Command:'' cd [HP-UX-directory-name] (CR) {The FTP command `cd' changes the working directory. E.g., `cd /tmp/user/[n]/{username] (CR)' changes to the user's temporary directory provided the proper disk number and id-number are used.}
``Command:'' put [UICVM-fn.ft.fm] [HP-UX-fn.ext] (CR) {The FTP `put' command (usually `send' for most FTP programs) stores the CMS file [UICVM-fn.ft.fm] into the UNIX file [HP-UX-fn.ext]. E.g., put craytest.fortran.e pgm.f transfers the test program from the class CMS disk to HP-UX. `mput' is the multiple version.}
``Command:'' ls (CR) {The FTP commands `ls}' and `dir' produce a working directory listing, with `ls' listing only file names and `dir' giving a full listing like `ls -lg'. Wild cards such as in `ls pgm.* (CR)' are permitted.}
``Command:'' quit (CR) {This command causes an exit form FTP and returns you to your CMS session. Caution: `bye' is not an alias for `quit' in the CMS FTP, as it is in UNIX FTP.}
``>>>QUIT ********''
``230 Goodbye.''
``MORE...'' Clear-key
``Ready; T=............'' {This Ready prompt indicates that you are back in CMS on the VM machine.}
An alternate method of sending files is to use the CMS NOTE command and reading in the CMS file using the CMS GET command. This BITNET file transfer method can produce variable results, because CMS SENDFILE does not work for this purpose, BITNET expects a blank first line in the message and it depends on all the computer links between here and there.
f90 -c +03 +list [source].f >& [source].LIST& (CR)
{Here, `+O3' enables parallel optimization and `+list' is the option that enables the helpful information marking of loops for optimization in the compiler information listing file that is automatically stored in the file `[source].LIST'. No additional option is needed for the automatic optimization, unless altered.}
f90 +O3 +U77 +list -o myrun [source].f >& [source].LIST & (CR)
{Here, the missing `-c' option means that both compiling and linking will be performed with an executable module produced rather than an object module and the additional option `-o [executable]' enables renaming of the executable to `myrun' for example rather as the default `a.out' executable name.
{For HP Standard C source file named `[source].c':}
cc -c +O3 [source].c (CR) {This is the HP Standard C compile command, with compiled source produced called `[source].o'. Here, the option `-c' denotes compilation, and `+O3' enables parallel optimization.
Also}
cc -o myrun [source].c >& [source].LIST & (CR)
{compiles and links the C program while producing messages on scalar and parallel optimizations.}
ld -o myrun [source].o (CR) {This step is the link or load step using the HP-UX segment loader with a typical format. Here the executable name is taken to be the generic name `myrun', but you can choose whatever name you like, omit the `-o' option produces the default UNIX name `a.out' for the executable. However, it is wise to stick to a single executable name, because the file size of typical executable is huge and a single name makes it easier to delete. It is usually cheaper to regenerate the executable than take up valuable storage space. }
{If you use UNIX redirection for input and output then the format is of the execute command is like:}
lsrun myrun < [data] >& [output] & (CR) {Here, `lsrun' is the LSF enveloping package command in the directory `/usr/local/lsf/bin/' (the Computer Center strongly urges you to use it, or else), `[data]' is the input file and `[output]' is the output file that also receives diagnostic messages in the background. Input files for Fortran `read' statements and output files for Fortran write statements can also be allocated to the terminal or HP-UX files using the Fortran `open' statement to reallocate units 5 and 6 as in the example programs used below.}
As practice, you can lsrun any source program that you have transported to HP-UX. {If available, the simple code `convert.f'
program convert
code: convert from debug fortran cogs, slightly modified.
change: input & output is to & from terminal, input at prompt.
Caution: compile, load, and execute in HP-UX using the three commands:
command: f90 -o convert convert.f
command: convert
real a(999)
write(*,*) 'input any integer less than 1000:'
read(*,*) i
a(i) = float(i)
write(*,6000) a(i)
6000 format(' floating point representation: ',e13.5)
write(*,*) 'What happens when you exceed array bound of 999?'
stop
end
Can also be obtained on UICVM CMS using}
GETDISK HANSON (CR)
and
filel convert f *
{Since this simple-minded `convert.f' program uses the terminal as undeclared input and output units, corresponding respectively to `read(*,*)' and 'write(*,[n])' statements, without specifying an the `open' statement. Be sure to do this in your temporary directory which you can change to by using `cd $TMP'. The source `convert.f' is executed with the 3 commands:}
f90 -o convert convert.f (CR)
convert (CR) {Upon execution with `convert', you are asked to supply an integer like 6:}
``Input an integer less than 1000:''
6 (CR) {HP-UX responds with the output:}
floating point representation: 0.60000E+01
What happens when you exceed array bound of 999?
STOP executed at line 15 in Fortran routine 'CONVERT'
CP: 0.002s, Wallclock: 2.311s
HWM mem: 163926, HWM stack: 2048, Stack overflows: 0
{To rerun the same code without recompiling, merely enter `convert' again:}
convert (CR) {Your response should be to enter another number as above. Do not spend too much time with `convert.f, because it can only read and write.}
The second example uses data files for both input unit 5 and output unit 6, as well as the UNIX Fortran seconds timer `second()'
CODE: Borg version (caution)
code_old: craytest fortran from cogs disk = ncsa ctss users guide eg#1
program tempt
calculation: c(i)=exp(-0.5)*pi/i
change: etime cpu timer replaces 'second'.
caution: note the use of a 2 subscript timer array is a fortran trick.
change: input from file 'tempt.data' and output to file 'tempt.output'.
change: particular input is the vector length "nx".
change: second cpu(user) time 'second' added.
caution: In HP9000 HP-UX compile, load and execute with:
command: f90 -o tempt tempt.f
command: tempt
parameter (ndim=5120)
real a(ndim),b(ndim),c(ndim)
real etime,tv(3,3)
caution: f90 "real" implies IEEE precision which means 23 or 24 bits fraction
caution: cft "real" implies 48bits = 6bytes for the fraction,
continued: unlike ibm "real" which implies 24bits = 3bytes.
continued: Otherwise all variables not starting with (i-n) are
continued: implicitly real, unless otherwise declared. The fraction
continued: for "double precision" is 96bits=12bytes, hence no "real*8"
open(6,file='tempt.output')
open(5,file='tempt.data',status='old')
read(5,*) nx
tv(3,1)=etime(tv(1,1))
tv(3,2)=etime(tv(1,2))
call init(ndim,nx,a,b)
call calc(ndim,nx,a,b,c)
tv(3,3)=etime(tv(1,3))
tuoh=tv(1,2)-tv(1,1) ! user cpu timer overhead
tsoh=tv(2,2)-tv(2,1) ! system cpu timer overhead
ttoh=tv(3,2)-tv(3,1) ! system cpu timer overhead
tusr=tv(1,3)-tv(1,2)-tuoh
tsys=tv(2,3)-tv(2,2)-tsoh
ttot=tv(3,3)-tv(3,2)-ttoh
write(6,66) nx,nx,c(nx),tuoh,tusr,tsoh,tsys,ttoh,ttot
66 format(1x,' nx=',i5,'; c(',i5,')=',f10.7
& /' ohead=',e12.4,' seconds; user time=',e12.4,' seconds',
& /' ohead=',e12.4,' seconds; system time=',e12.4,' seconds',
& /' ohead=',e12.4,' seconds; total time=',e12.4,' seconds')
stop
end
subroutine init(ndim,nx,a,b)
comment: removed cdir$ novector
real a(ndim),b(ndim) pi=acos(-1.0) do 10 ix=1,nx
a(ix)=pi b(ix)=float(ix)
10 continue
return end
subroutine calc(ndim,nx,a,b,c)
comment: removed cdir$ novector
real a(ndim),b(ndim),c(ndim) do 20 ix=1,nx
c(ix)=exp(-0.5)*a(ix)/b(ix)
20 continue
return
end
{The source is a modified version of the old USER'S GUIDE craytest code. A copy can be obtained via a web browser:
or a copy can be obtained by Anonymous FTP:
or
An old Cray copy also resides in Hanson`s public CMS disk. Use your own copy or get Hanson`s CMS copy by using HP-UX `ftp' to your own UICVM CMS account and when in FTP, enter
``ftp>'' cd u12688.192.r (CR)
``ftp>'' get tempt.f (CR)
``... {return code messages} ...''
``ftp>'' bye (CR) {`bye' is often an FTP alias for `quit', but not in IBM FTP. 'tempt.f' performs a HP `real', 24 bit precision, floating point vector calculation and times the calculation using `second()' It requires that a single vector dimension (not more than 5120) be inputted from an unformatted file called `tempt.data' using a `read(5,*)' statement, while the output is written to a file called `tempt.output' using a `write(6,[n]) [list]' statement. These files, usually enclosed in quotes, are allocated units with the `open' statements `open(5,file='tempt.data')' and `open(6,file='tempt.output')'. Before executing `tempt.f', a file called `tempt.data' is required. You can use the HP-UX line editor `ex' to create one as follows:}
``borg:'' ex tempt.data (CR) {Caution: you should not use `ex' or `vi' for extensive editing on the HP9000, which you can do at your local host, UICVM or UNIX; however, this short editing session will only take a short amount of time. A selected list of `ex' commands are given below. The `ex' prompt is `:'. You add lines with the `ex' subcommand `0a' to start with, to add one after line 0 in:}
``:'' 0a (CR)
500 (CR) {Note that using the IBM Telnet interface, you enter the data at the bottom of the screen and only when you return does it appear as above. End the insert mode by a entering a solitary period:}
. (CR) {Finally save and exit `ex' with the combined `wq' = `w | q' subcommand:}
``:'' wq (CR) {If successful (you can check by `cat tempt.data (CR)' you should now have files called `tempt.data' from `ex' and `tempt.f' from `ftp' or other source, so that you can now compile, load, and execute `tempt.f' by entering the three command lines:}
``borg:'' f90 -o tempt tempt.f (CR)
``borg:'' lsrun tempt (CR)
{Your output will be in `tempt.output' and you can list is by the command:}
``borg:'' cat tempt.output (CR) {Your output should look something like:}
nx= 500; c( 500)= .0038109 ohead= .0000E+00 seconds; user time= .0000E+00 seconds ohead= .0000E+00 seconds; system time= .1000E-01 seconds ohead= .0000E+00 seconds; total time= .1000E-01 seconds
{If you wish to re-run the program again with a different number, the enter}
``borg: '' ex tempt.data (CR) {again, or}
``borg: '' !ex (CR) {and enter within EX the subcommand}
``:'' 1c (CR)
5000 (CR)
. {to change `500' to `5000' and to end the change subcommand, while to end `ex' enter after the ``:'' prompt:}
``:'' wq (CR) {after the ``:'' prompt, and then enter}
``borg: '' lsrun tempt (CR) {and}
``borg: '' cat tempt.output (CR) {again. When you are done with `tempt.f', remove all the files from HP-UX that you do not need, using:}
``borg: '' rm tempt (CR) {for example, but especially the big executables like `tempt'.}
For information on C language programs use the HP-UX commands:
man cc (CR) : for the manual help pages on the C compile and load command; also the `-q' option of man will produce quick, compact manual pages.
On the `borg' enter
f90 +list -[options] [source].f [other source files] >& [source].LIST& (CR) : Compiles and links source file `[source].f' and `[other source files]' with the HP compiler listing option `+list' redirecting listing and error message output to `[source].LIST' in the background. The actual compiler is called `fskel', but the compiler controller `f90' automatically calls it. The command
f90 -c +list [source].f >& [source].LIST& (CR) : Only compiles {`-c}' the source file `[source].f' and yields a compiler optimization report, with default optimization level `O2'.
Use other options
f90 +U77 +list -[options] -o [executable] [source].f >& [source].LIST & (CR) : The `f90' with option `-o [executable]' produced an named executable file rather than the default `a.out' executable file.
Note: It is much better to use makefiles for such commands.
The current optimizing HP-UX C compiler is `/usr/convex/cc6.5/cc'.
cc -[options] -o myrun [file].c (CR) : Compiles source [file].c, using the standard C compiler `cc6.5' and producing an executable named `myrun'. Use options {similar to Fortran `f90' compiler list}
`+O0' for basic block scalar optimizations,
`+O1' for program unit scalar optimizations plus `O0',
`+O2' for global instruction scheduling, software pipelining, and data localizations plus `O1' {`+O2' = `-O' is the default optimization level},
`+O3' for full parallel optimizations plus `O2',
`+O4' for link time parallel optimizations plus `O3',
`+O[optlevel] +[optoption] for required optimization level and other optimization options,
`-o [executable-file]' to name the executable module instead of the `a.out' default,
`-c' for compilation only {no link or executable} and `[source].o' binary object file {`ld' command to is then needed to load the execution module, which then can be used to execute the program.}
Use `man cc' for more about options.
cc -c [file].c (CR) : Compiles source [file].c, using the standard C compiler `cc6.5' and producing an object file named [file].o.
cc +L -o [executable_file] [file].c >& [file].LIST & (CR) : Produces an optimization report including all tables, which may be restricted by replacing `all' by `loop', `private' or `array', the report itself is redirected to the output file `[file].LIST' in the background.
See `man cc' for more information.
#pragma [_CNX] [directive] : Form of C compiler directive placed within the C code, where some example directives are `LOOP_PARALLEL' for running next loop in parallel, `PARALLEL([list])' and `END_PARALLEL' for marking a code segment for parallel execution, `PARALLEL_PRIVATE([list])' for designating a list of variable in a parallel code segment as private or local. Refer to `man cc' or the `C User's Guide' for more information.
See `man ld' for more information. Note that the linker or loader `ld' works for both Fortran and C code object files.
ld -o [executable-file] -l [library list] [source].o (CR) : This segment loader links and loads the object module `[source].o' from the `f90' step into the execution module named `[executable-file]' by the `-o' option. Without the `-o' option, the executable is the standard `a.out' file. The library option may not be needed because many libraries are searched by default: Fortran90 (/usr/convex/fc9.5/lib/libF90.a), C (/lib/libc.a), I/O (/usr/lib/libio.a) and Math (/lib/libm.a).
To find out what other special software is at UIC click on:
lsrun [executable-file] < [input-file] > [output-file] & (CR) : Executes the executable module taking input from the file `[input-file]' and redirecting output to `[output-file]' as a background process.
The HP HP9000 has several Parallel Information Functions that permit finding the number of processors with threads (parallel execution streams running on parallel processors), the number of threads, and similar information. The type declarations and usage of these functions is best illustrated by the following HP Fortran `f90' code fragments:
Fortran Example:
c ............ deleted nonrelevant code
Check: Parallel Information Functions
Caution: Avoid "()" args in declaration statements as in Programmer's Guide.
Check: Output (write) statements for function name meanings.
Code: Typical Parallel Information Function Declaration Statements:
integer num_procs
integer num_threads
integer num_nodes
integer num_node_threads
integer my_thread
integer my_node
integer level_of_parallelism
Code: Typical Parallel Information Output Variable Declaration Statements:
integer nproc,nth,nnode,nnodeth,myth,mynode,levpar,vlevpar(n)
c ............ deleted nonrelevant code
Code: Typical Parallel Information Function Output Statements:
write(6,*) 'Parallel Information Function Output:'
606 format(1x,a,' = ',i3)
Caution: Deadlock could result if num_procs, etc., are used in I/O operations,
continued: i.e., do not use Parallel Information Functions directly in writes.
nproc=num_procs()
write(6,606) 'Number Processors with Threads',nproc
nth=num_threads()
write(6,606) 'Number Threads',nth
nnode=num_nodes()
write(6,606) 'Number HyperNodes',nnode
nnodeth=num_node_threads()
write(6,606) 'Number Threads on HyperNodes',nnodeth
myth=my_thread()
write(6,606) 'My Thread ID',myth
mynode=my_node()
write(6,606) 'My Hypernode ID',mynode
levpar=level_of_parallelism()
write(6,606) 'Level of Parallelism',levpar
Code: Output Table Explanation of Allowed Levels of Parallelism 0 to 9:
write(6,*) 'where level 0 means Not Parallel'
write(6,*) 'where level 1 means Asymmetric Thread Parallelism'
write(6,*) 'where level 2 means Node Parallelism'
write(6,*) 'where level 3 means Node Parallelism plus level 1'
write(6,*) 'where level 4 means Thread Parallelism'
write(6,*) 'where level 5 means Thread Parallelism plus level 1'
write(6,*) 'where level 6 means Thread and Node Parallelism'
write(6,*) 'where level 7 means Thread Parallelism plus levels',
& '1 & 2'
write(6,*) 'where level 8 means Single Dim. Thread Parallelism'
write(6,*) 'where level 9 means Single Dim. Thread Parallelism',
& 'plus level 1'
c ........ deleted nonrelevant code:
Code: Example of Use of "level_of_parallelism()" in a loop:
do 5 i=1,n
.........
CodeTechnique: vector form is used to be less likely to hinder loop parallelism:
vlevpar(i)=level_of_parallelism()
.........
5 continue
write(6,606) 'Level of Parallelism: do 5',vlevpar(n)
c.............
.......
C Example (untested):
/* Include Parallel Information Required Header File: */
#include
Some command options can not be directly entered in compiler, load or execution commands, but the `mpa' modifying attribute commands permits specifying such things as the number of processors or threads for an executable module file.
mpa -[options] [executable_file] (CR) : Modifies the attributes of an executable `[executable_file]' according to the options `-[options]'. For example,
mpa -min 2 -max 4 -over -m myrun & (CR) : Permits the use of between a minimum of 2 and a maximum of 4 processors or threads when executing `myrun', while allowing oversubscription {`-over'} of threads with the modification {`-m'} of the given file `myrun'.
mpa -min 8 -max 8 -over -m myrun & (CR) : Permits exactly 8 processors or threads to be used. {Caution: the Computer Center does not allow `borg' to be used with more than 8 processors without special permission; also timing for of each execution is necessary since more processors does not necessarily mean faster execution due to overhead considerations. Also, since the `borg' is configured into processor sets called subcomplexes, typically as SC=sys with four processors, SC=c4 with four processors and SC=c8 with 8 processors, it would be wise to have jobs, using more than a few threads, running in a single subcomplex with a sufficient number of physical processors. The current subcomplex configuration is determined by the command:}
scm -c (CR) : Subcomplex manager command list the current subcomplexes, their node IDs, subcomplex names, subcomplex IDs and processor IDS within each subcomplex. For Example,
mpa -sc c8 -min 6 -max 8 -over -m myrun & (CR) : Permits between 6 and 8 processors to be used and to be assigned from the subcomplex `c8' with 8 processors.
Use `man mpa' for more information.
f90 -cdxb -ctifiles -o myexec [source].f (CR)
cxdb -ctifiles -o myexec (CR) : Compiles the Fortran program [source].f for using the HP CXDB debugging facility with debugging information placed in a Compiler Tools Interface (CTI) directory `.CTI'.
See also the HP `prof' profiling command using `man prof'.
qsub [options] (CR) : Submit a batch job to the queue; see `man qsub (CR)' for more information. The option, for example, `-lM [16Mw]' permits running jobs with up to 16 mega words of memory, for example. The option `[myjob].script' provides the script instructions for running a background job. Note that UIC users must specify a script line
See also the UIC Exemplar Home Page section
for more information on the NQS queues.
qstat [options] (CR) : Display status of queued batch jobs; see `man qsub (CR)' for more information.
For more information about batch processing with NQS, check the man pages for
qdel
qps
qstat
HP Fortran90 (f90) contains extensions beyond `f77' Fortran. Fortran90 is best used from within the HP Fortran optimizing compiler `f90' (Although you can not put the `-f90' option on the `f90' command, the optimization report for `f90 +O3 +list' shows that `f90' is indeed the implicit default even if not requested, although it may be turned off with the `-nof90' option.
The `f90' compiler is part of the Pacific-Sierra Research Corporation VAST optimizing compiler that is used by most optimizing compilers and is used along with the `vf90' translator, both being found in the directory `/usr/convex/vast90/' {CAUTION: There is no `vf90' license on the `borg' now and `f90' needs `vf90'}.
For optimization, it is recommended that your f90 program aid the f90 parallel optimization model, i.e.,
See also Section
and Subsection
Also see the appropriate sections, `man cc' for items on HP Standard C.
Fortran 90 Array Notation and Array Sections: HP Fortran f90 allows most Fortran90 extensions for arrays, making array statements like
C = A + B ! arrays A, B, C must have the same ``shape'' (dimensions); i.e., same as `C(i)=A(i)+B(i)' with in a do loop for all `i' of subscripts.
A = 3*B + S ! for scalar S and array A; exception to same ``shape'' rule. i.e., same as `A(i)=3*B(i)+S' with in a do loop for `i=1,N', N=[dimension].
A(1:50) = B(1:100:2)' ! array sections must have same ``shape'', i.e., same as `A(i)=B(2*i-1))' for `i=1,50' in a do loop.
A([start]:[end]:[step])=[expression] ! in general, references the single subscript array section for i = [start] to [end] in steps of [step].
a(i,:)=[expression] ! for the i-th row of array `a'.
a(:,j)=[expression] ! for the j-th column in `a'.
a(1::2)=[expression] !for the odd vector elements.
a(n:1:-1)=[expression] ! for the `n' vector elements of `a' in reverse order.
z(1:n) = -log(z(1:n)) ! example for Fortran built-in function.}
Array Constructors: Fortran 90 array constructors permit initialization of vectors and arrays by enclosing data separated by commas with all data enclosed between `(/' and `/)' delimiters. For example (assuming proper dimensioning):
a=(/1,2,3,4,5,6/) ! for `integer a(6)'; `!' delimits inline comments
a=(/(i,i=1,6)/) ! do loop form, enclosed by `(' and `)'
b(1,:)=(/1,3,5/) ! for first row when `integer b(2,3)'
b(2,:)=(/(2*i,i=1,3)/) ! do loop form of constructor for 2nd row of `b'
Caution: Constructors can not be currently used in print statement arguments.}
real [variables-list] {The f90 `real' declaration declares variables and array elements as 32-bit (4-byte) words with only 23-bits (6-bytes) allotted to the fraction for precision with one additional bias bit. This is IEEE precision, and is half of the precision on the Cray T90 YMP. The f90 `double precision' or `real*8' declaration is 64-bits with a 53-bit fraction, and hence is entirely different from Cray 128-bit `double precision'.}
POINTER (P,A) {The f90 new Fortran `pointer' statement declares that the declared integer (usually) variable holds (points to, for C-fans) the shifted initial (base) address of the declared array A. Not tested.}
open ([unit],file=`[fn]',status='unknown') {Format of f90 OPEN statement assigning unit number [unit] to filename [fn]; place in program after declarations;
[unit] = 5 defaults to UNIX `stdin' as does
[unit] = * for read statements or reads from the terminal unless it is redirected by an `open' or a `lt;';
[unit] = 6 defaults to UNIX `stdout' as does
[unit] = * for write statements or writes to the terminal unless it is redirected by an `open' or a `>';
[unit] = 0 defaults to UNIX `stderr' or writes diagnostics to the terminal unless it is redirected by an `open' or a `>&'; note that file names are placed in quotes in the OPEN statement; see also `man' for HP-UX `assign' and `env' statements.}
save [variable or array name list separated by commas] {The save statement is essential in f77 subroutines to save parameter variable values for later calls to a subroutine; the `-ev' option of f90 provides a better solution to this problem; if not used can lead logic errors, especially for users accustomed to F66 Fortran in which variables are saved after the RETURN statement is executed, but lost in f77.}
recursive [function or subroutine]([subprogram arguments]) {The 'recursive' prefix is required on subprograms called recursively, but also the recursive suboption is needed in the compiler statement.}
[statement] ! [embedded comment] {The line embedded comment is now legal in HP Fortran.}
intrinsic [f90-function1][,[f90-function2]] {An Intrinsic statement is needed in `f90' to declare any to Fortran90 extended intrinsics, such as ANY, DOT_PRODUCT, MAXVAL, ALL, EOSHIFT, MINLOC, SPREAD, COUNT, FLOAT, MINVAL, SUM, CSHIFT, MATMUL, PACK, TRANSPOSE, MAXLOC, PRODUCT, UNPACK.}
Many of the Fortran90 Array Functions that have been available on the Connection Machine are now available for the `f90' compiler version 9.5. However, a few functions take their arguments in a different order on the HP than on the Connection Machine.
intrinsic pack, unpack, spread {Reminder that F90 extended intrinsic functions need an `intrinsic' statement right after the type declarations. }
PACK([array],[mask-array][,[vector]]) {Transforms (packs) the array `[array]' into a vector `[vector]' (an optional argument, which if not present, the output goes to the value of the function) according to the true values of the `[array]'-conformable, logical mask `[mask-array]'. }
UNPACK([vector],[mask-array],[field-array]) {Transforms (unpacks) the vector `[vector]' into the array `[field-array]' according to the true values of the `[field-array]'-conformable, logical mask `[mask-array]'. }
SPREAD([array],[dim],[ncopies]) {Transforms (spreads) the source array `[array]' into the output value of the function with `[ncopies]' copies along the dimension `[dim]' (horizontal copies if `[dim]'=1 and vertical if `[dim]'=2. }
RESHAPE([array],[shape][,[pad]][,[order]])
{Transforms (reshapes) the source array `[array]' into the output value of the
function with shape `[shape]' with order `[order]' padding the array `[pad]'.
Caution: `reshape' has not yet been implemented in `HP f90'.
}
srand48([iseed])
[uniform]=drand48()
{Generates pseudo random numbers uniformly distributed with the value in
the `uniform_variable' argument and is part of the standard HP Fortran
and C `stdlib.h' libraries rather than for f90, but the double precision
`drand48' is recommended over the simple `rand' utility.
The utilities `srand48' and `drand48' must be used in the sequence indicated.
The value of the argument seed `[iseed]' must
be nonzero to start a new random sequence.
}
call random_number([uniform_variable])
{Should generate pseudo-random numbers uniformly distributed with the
value in the `uniform_variable' argument, which is called the
`harvest' tag.
Caution: DOES NOT WORK WITH `f90' even with an `intrinsic'
statement: use the standard `drand48' (see `man drand48' for information
on `drand48' and similar pseudo-random number generators and the example
here in the f90test code).}
call random_seed([SIZE=[seed_size]][PUT=[seed](1:[seed_size])]
[GET=[seed](1:[seed_size])])
{For `random_number', should either set random seed vector size, or put
(sets) the seed, or gets the seed.
Caution: see above.
}
The reduction functions reduce the input to a scalar output.
intrinsic sum, product, spread, maxval, minval, count, any, all {Reminder that F90 extended intrinsic functions need an `intrinsic' statement. }
SUM([array][,[dim][,[mask]]]) {The `SUM' function computes the sum of the elements of the array `[array]' along the dimension `[dim]' (by columns if `[dim]'=1 or by rows if `[dim]'=2 or `[dim]'=0 for the whole array), according to the true values in the conditional mask `[mask]', if present. This function makes the HP sum function the same as the Connection Machine version except that `dim=[dim]' and `mask=[mask]' labels are illegal in `HP f90',as they are in the rest of the functions in this section. }
PRODUCT([array][,[dim][,[mask]]]) {The `PRODUCT' function computes the product of the elements of the array `[array]' along the dimension `[dim]' (by columns if `[dim]'=1 or by rows if `[dim]'=2 or `[dim]'=0 for the whole array), according to the true values in the conditional mask `[mask]', if present. }
MAXVAL([array][,[dim][,[mask]]]) {The `MAXVAL' function computes the maximum value of the elements of the array `[array]' along the dimension `[dim]' (by columns if `[dim]'=1 or by rows if `[dim]'=2 or `[dim]'=0 for the whole array), according to the true values in the conditional mask `[mask]', if present. }
MINVAL([array][,[dim][,[mask]]]) {The `MINVAL' function computes the minimum value of the elements of the array `[array]' along the dimension `[dim]' (by columns if `[dim]'=1 or by rows if `[dim]'=2 or `[dim]'=0 for the whole array), according to the true values in the conditional mask `[mask]', if present. }
COUNT([mask][,[dim]]) {The `COUNT' function computes the number of the true elements of the logical array `[mask]' along the dimension `[dim]' (by columns if `[dim]'=1 or by rows if `[dim]'=2 or `[dim]'=0 for the whole array), if present. }
ANY([mask][,[dim]]) {The `ANY' function computes if there are any true elements in the logical array `[mask]' along the dimension `[dim]' (by columns if `[dim]'=1 or by rows if `[dim]'=2 or `[dim]'=0 for the whole array), if present, and returns a logical true or false answer. }
ALL([mask][,[dim]]) {The `ALL' function computes if there are all true elements in the logical array `[mask]' along the dimension `[dim]' (by columns if `[dim]'=1 or by rows if `[dim]'=2 or `[dim]'=0 for the whole array), if present, and returns a logical true or false answer. }
The manipulation functions rearrange the elements of the target matrix. See `cm-guide.tex' on `CMS getdisk hanson' for examples. However, the arguments may be in a different order on the Connection Machine.
intrinsic transpose, eoshift, cshift {Reminder that F90 extended intrinsic functions need an `intrinsic' statement. }
TRANSPOSE([array]) {The `TRANSPOSE' function transposes the 2-subscript array `[array]' with the result array of reversed dimensions. `transpose' is useful for converting arrays prior to printing since the transposed array will be printed out by rows, since the original array would be printed out by columns due to the column-wise nature of Fortran by default when array notation is used in a print statement. Note: `man transpose' leads to the `LIBF90' manual page, but `man libf90' does not. }
EOSHIFT([array],[shift][,[boundary][,[dim]]]) {The `EOSHIFT' function does an end-off shift on the array `[array]' along the dimension `[dim]' using the boundary value(s) `[boundary]' to fill in, if necessary. Caution: `man eoshift' does not lead to documentation, indicating it is not officially supported, but tests below show it works. Connection Machine arguments have a different order. }
CSHIFT([array],[shift][,[boundary][,[dim]]]) {The `CSHIFT' function does a circular shift on the array `[array]' along the dimension `[dim]' using the boundary value(s) `[boundary]' to fill in, if necessary. Caution: man eoshift' does not lead to documentation, indicating it is not officially supported, but tests below show it works. Connection Machine arguments have a different order. }
The location functions find the location of elements of the target matrix. See `cm-guide.tex' on `CMS getdisk hanson' for examples. However, the arguments may be in a different order on the Connection Machine and the Connection Machine also has the location functions `firstloc', `lastloc' and `project'.
intrinsic maxloc, minloc {Reminder that F90 extended intrinsic functions need an `intrinsic' statement. }
MAXLOC([array][,[mask]]) {The `MAXLOC' function finds the first element of target array `[array]' having the maximum value, relative to the conditional mask `[mask]', if present. }
MINLOC([array][,[mask]]) {The `MINLOC' function finds the first element of target array `[array]' having the minimum value, relative to the conditional mask `[mask]', if present. }
The matrix multiply functions compute the matrix products of the target matrices. See `cm-guide.tex' on `CMS getdisk hanson' for examples.
intrinsic matmul, dot_product {Reminder that F90 extended intrinsic functions need an `intrinsic' statement. }
MATMUL([array1][array2]) {The `MATMUL' function computes the matrix product of target arrays `[array1]' and `[array2]' commensurate for multiplication, with the result matrix of appropriate size. This function is also used for matrix-vector multiplication. }
DOT_PRODUCT([vector1][vector2]) {The `DOT_PRODUCT' function computes the scalar, dot product of target vectors `[vector1]' and `[vector2]', with the scalar result. Caution: the Connection Machine function is `dotproduct'. }
The following f90 code contains examples of use of many of the Fortran90 array intrinsic functions mentioned above. There are some subtle differences:
If b = 1 3 5 logical mask=b.gt.3
2 4 6
then s3=sum(b,1,mask) or s2=sum(b,2,mask) work when
real s3(3),s2(2)
but isum=sum(b,mask) or isum=sum(b,,mask) or isum=sum(b,:,mask) do NOT work.
That is, how do you enter a scalar dim for the whole array?
borg: f90 +O3 +U77 +list -o f90test f90test.f >& f90test.LIST&
borg: lsrun f90test >& f90test.output&
%%%%%%%%%%% begin f90test.f %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
program f90test
code98: use drand48() pseudo random number generator since random_number bad
code97: update by removing old comments to cmfortran
code96: retest=f90test.f redone on borg = convex spp1200/xa-16
cf90er: => convex fc9.5 flagged for error.
integer, parameter :: m = 6
integer, parameter :: n = 4
integer :: i,j
integer :: seedval
integer, dimension(2) :: s2, ctr1, ctr2, ctr3, b2 ,cr1 ,cr2
integer, dimension(3) :: s3 ,at ,ar1 ,ar2 ,br1 ,br2
integer, dimension(4) :: as(4)
integer, dimension(2,2) :: c ,bi
integer, dimension(2,3) :: b, a ,cu
integer, dimension(3,2) :: ct
integer, dimension(3,4) :: cs
integer, dimension(4,3) :: cst
logical, dimension(2,3) :: test
logical, dimension(64,64) :: inmask
double precision, parameter :: tol = 0.5e-5
integer, parameter :: niter = 5000
double precision :: diffav
double precision, dimension(8,8) :: us
double precision, dimension(64,64) :: u , du
double precision :: drand48
double precision, dimension(m,n) :: uniran
double precision, dimension(n,m) :: truniran
intrinsic sum,maxval,minval,product
& ,dot_product,matmul,transpose
& ,cshift,eoshift,spread,random_number
data b/1,2,3,4,5,6/ !replace constructors initialization
data as/2,3,4,5/
data at/2,3,4/
c --------------------Array Constructors:
b(1,1:3) = (/1, 3, 5/) ! initialize first row, along dimension 2.
b(2,1:3) = (/2, 4, 6/) ! initialize second row, along dimension 2.
print*,'Note: constructors like "(/1,2/)" allowed in fc9.5'
br1 = b(1,:)
br2 = b(2,:)
print60,br1,br2
60 format(' b(2,3)'/(3i3))
c --------------------Sum Function sum:
isum = sum(b) ! => isum = 21; i.e., Front-End scalar.
print61,' isum=sum(b)=',isum
61 format(1x,a36,i4)
isum = sum(b(:,1:3:2)) ! => isum = 14; sole ':' means all values '1:2'.
print61,' isum = sum("b(:,1:3:2)")=',isum
bi=b(:,1:3:2)
isum=sum(bi)
print61,' isum = sum("b(:,1:3:2)")=',isum
print*,'CAUTION: "dim=", etc., markers= NOT allowed in intrinsics'
s2 = sum(b,2) ! redeclared with the correct array section shape.
print62,' s2 = sum(b,2)=',s2 ! => s2 = (/9,12/), row sums
62 format(1x,a32,2i3)
s3 = sum(b,1) ! => s3 = (/3,7,11/); column sums.
print63,' s3 = sum(b,1)=',s3
63 format(1x,a32,3i3)
print*,'CAUTION: "mask=" marker= STILL not allowed either.'
s3 = sum(b,1,b.gt.3) ! => s3 = (/0,4,11/); i.e., conditional col sum
print63,' s3 = sum(b,1,"b.gt.3") =',s3
test=b.gt.3
s3 = sum(b,1,test) ! => s3 = (/0,4,11/); i.e., conditional col sum
print63,' s3 = sum(b,1,"b.gt.3") =',s3
s2 = sum(b,2,test) ! => s2 = (/5,10/); i.e., conditional row sum
print62,' s2 = sum(b,2,b.gt.3) =',s2
cf8er:isum = sum(b,0,test) ! => isum = 18; i.e., add only elements
cf8er:print61,' isum = sum(b,0,b.gt.3) =',isum ! that are greater than three.
print*,' CAUTION: If "sum(array[dim[,mask]])", CANT use zero (0)'
& ,' for [dim] for whole array when there is a mask.'
c --------------------Maximum Value Function maxval:
imax = maxval(b) ! => imax = 6; array maximum value.
print61,' imax = maxval(b)=',imax
s3 = maxval(b,1) ! => s3 = (/2,4,6/); column maximums.
print63,' s3 = maxval(b,1)=',s3
s2 = maxval(b,2) ! => s2 = (/5,6/); row maximums.
print62,' s2 = maxval(b,2)=',s2
c --------------------Minimum Value Function minval:
imin = minval(b) ! => imin = 1; array minimum value.
print61,' imin = minval(b)=',imin
c --------------------Product Function product:
s2 = product(b,2) ! => s2 = (/15,48/); products of column elements.
print62,' s2 = product(b,2)=',s2
c --------------------Dot Product Function dot_product:
idot = dot_product(br1,br2) ! => idot = 44; dot product of row
print61,' idot = dot_product(b(1,:),b(2,:))=',idot ! vectors of b.
print*,' CAUTION: Array syntax not allowed in actual arguments.'
c --------------------Matrix Multiplication Function matmul:
! assuming array b of the previous section.
![Ans] = matmul([Array_1],[Array_2]) ! computes matrix multiplication
! of two rank two matrices.
c = matmul(b(:,1:2),b(:,2:3)) ! => c(1,:)=(/15,23/);c(2,:)=(/22,34/).
c=transpose(c)
print623,'c=matmul(b(:,1:2),b(:,2:3))=',c
623 format(1x,a36/(2i3))
![Ans] = transpose([Array]) ! transforms an array to its transpose.
ct = transpose(b) ! => ct(1,:)=(/1,2/);ct(2,:)=(/3,4/);ct(3,:)=(/5,6/).
ctr1 = ct(1,:)
ctr2 = ct(2,:)
ctr3 = ct(3,:)
print623,'ct = transpose(b)=',ctr1,ctr2,ctr3
c --------------------Circular Shift Function cshift:
! assume b is again initialized as
! b = 1 3 5
! 2 4 6
a = cshift(b,1,2) ! => a = 3 5 1
! 4 6 2
cshift EG1:
ar1 = a(1,:)
ar2 = a(2,:)
print633,'a = cshift(a,1,2)=',ar1,ar2
633 format(1x,a36/(3i3))
! i.e., b(i,(j+shift) "mod" n) -> a(i,j) for j=1:2, etc.;
! nonstandard modulus fn: 0 "mod" n = n; 1 "mod" n = 1; ...; n "mod" n = n
! i.e., the result is computed from shifting subscript in specified
! dimension of the source array by the specified shift.
a = cshift(b,-1,2) ! => a = 5 1 3
! 6 2 4
cshift EG2:
ar1 = a(1,:)
ar2 = a(2,:)
print633,'a = cshift(b,-1,2)=',ar1,ar2
! i.e., b(i,(j+shift) "mod" n) -> a(i,j) for j=2:3, etc.
cshift EG3:
s2(1) = 1
s2(2) = 2
a = cshift(b,s2,2) ! a = 3 5 1
! 6 2 4
! i.e., an array-valued shift, or shift per row.
ar1 = a(1,:)
ar2 = a(2,:)
print633,'a = cshift(b,(/1,2/),2)=',ar1,ar2
cshift Laplace Example:
! Jacobi Iteration for a 5-star discretization of
! 2D Laplace's equation:
u = 0
u(1,:)=2
u(64,:)=2
u(:,1)=2
u(:,64)=1
inmask = .FALSE.
inmask(2:63,2:63) = .TRUE.
diffav = 1
iter=0
do while (diffav.gt.tol.and.iter.lt.niter)
iter=iter+1
du = 0
where(inmask)
du = 0.25*(cshift(u,1,1)+cshift(u,-1,1)+cshift(u,1,2)
& +cshift(u,-1,2)) - u
u = u + du
end where
du = du*du
diffav = sqrt(sum(du)/(62*62))
end do
! which is the main program fragment of laplace.fcm.
cf90er:print66,'u = laplace-shift(u)=',u(1:64:16,1:64:16)
cf90er: & ,' array section like "u(1:64:16,1:64:16)".'
print*,'CAUTION: array sections not allowed in print'
us = u(1:64:9,1:64:9)
us=transpose(us)
print66,'u = laplace-shift(u)= ; iter=',iter,'; av-diff ='
& ,diffav,us
66 format(1x,a36,i5,a11,e10.3/(8f8.4))
c --------------------End Off Shift Function eoshift:
a = eoshift(b,-1,0,1) ! a = 0 0 0 note default boundary value is 0.
! 1 3 5
ar1 = a(1,:)
ar2 = a(2,:)
print633,'a = eoshift(b,-1,0,1)=',ar1,ar2
s2=(/-1,0/)
b2=(/7,8/)
a = eoshift(b,s2,b2,2) ! => a = 7 1 3
! 2 4 6
ar1 = a(1,:)
ar2 = a(2,:)
print633,'a = eoshift(b,(/-1,0/),(/7,8/),2)=',ar1,ar2
a = eoshift(b,2,0,2) ! => a = 5 0 0
! => 6 0 0
ar1 = a(1,:)
ar2 = a(2,:)
print623,'a = eoshift(b,2,2)=',ar1,ar2
c --------------------Spread Function spread:
cs = spread(as,1,3)
! contents of cs:
! 2 3 4 5
! 2 3 4 5
! 2 3 4 5
cst = transpose(cs)
print64,'as =',as
64 format(1x,a32,4i3)
print643,'cs = spread(as,1,3)=',cst
643 format(1x,a36/(4i3))
c --------------------
cs = spread(at,2,4)
! contents of c:
! 2 2 2 2
! 3 3 3 3
! 4 4 4 4
cst = transpose(cs)
print63,'at =',at
print643,'cs = spread(at,2,4)=',cst
c ---------------------------------------------------------------------------
! i.e., b=spread(a,d,c) =>
! a(n_1,n_2,...,n_(d-1),n_d,...,n_r) -> b(n_1,n_2,...,n_(d-1),c,n_d,...,n_r)
! where r is the rank of source array a and n_i is the size of dimension i;
! noting that a new dimension of size c is added before dimension d.
c ---------------------------------------------------------------------------
C Random Number Generator: NOT random_number, BUT drand48 with srand48
c random_number executes, but gives nonrandom arrays: use drand48() instead
print*,'F90 random_number still does work not on borg, Fall 1998'
seedval = 123456
call srand48(seedval)
do i = 1, m
do j = 1, n
uniran(i,j) = drand48()
enddo
enddo
truniran = transpose(uniran)
write(6,65) truniran
65 format(' function drand48() uniform random array:'/(4f14.10))
stop
end
%%%%%%%%%%% end f90test.f %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%% begin f90test.output %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Note: constructors like "(/1,2/)" allowed in fc9.5
b(2,3)
1 3 5
2 4 6
isum=sum(b)= 21
isum = sum("b(:,1:3:2)")= 14
isum = sum("b(:,1:3:2)")= 14
CAUTION: "dim=", etc., markers= NOT allowed in intrinsics
s2 = sum(b,2)= 9 12
s3 = sum(b,1)= 3 7 11
CAUTION: "mask=" marker= STILL not allowed either.
s3 = sum(b,1,"b.gt.3") = 0 4 11
s3 = sum(b,1,"b.gt.3") = 0 4 11
s2 = sum(b,2,b.gt.3) = 5 10
CAUTION: If "sum(array[dim[,mask]])", CANT use zero (0) for [dim] for whole array when there is a mask.
imax = maxval(b)= 6
s3 = maxval(b,1)= 2 4 6
s2 = maxval(b,2)= 5 6
imin = minval(b)= 1
s2 = product(b,2)= 15 48
idot = dot_product(b(1,:),b(2,:))= 44
CAUTION: Array syntax not allowed in actual arguments.
c=matmul(b(:,1:2),b(:,2:3))=
15 23
22 34
ct = transpose(b)=
1 2
3 4
5 6
a = cshift(a,1,2)=
3 5 1
4 6 2
a = cshift(b,-1,2)=
5 1 3
6 2 4
a = cshift(b,(/1,2/),2)=
3 5 1
6 2 4
CAUTION: array sections not allowed in print
u = laplace-shift(u)= ; iter= 4730; av-diff = 0.499E-05
2.0000 2.0000 2.0000 2.0000 2.0000 2.0000 2.0000 1.0000
2.0000 1.9762 1.9479 1.9090 1.8491 1.7440 1.5208 1.0000
2.0000 1.9573 1.9068 1.8387 1.7387 1.5836 1.3402 1.0000
2.0000 1.9469 1.8844 1.8014 1.6836 1.5141 1.2817 1.0000
2.0000 1.9469 1.8844 1.8014 1.6836 1.5141 1.2817 1.0000
2.0000 1.9573 1.9068 1.8387 1.7387 1.5836 1.3402 1.0000
2.0000 1.9762 1.9479 1.9090 1.8491 1.7440 1.5208 1.0000
2.0000 2.0000 2.0000 2.0000 2.0000 2.0000 2.0000 1.0000
a = eoshift(b,-1,0,1)=
0 0 0
1 3 5
a = eoshift(b,(/-1,0/),(/7,8/),2)=
7 1 3
2 4 6
a = eoshift(b,2,2)=
5 0
0 6
0 0
as = 2 3 4 5
cs = spread(as,1,3)=
2 3 4 5
2 3 4 5
2 3 4 5
at = 2 3 4
cs = spread(at,2,4)=
2 2 2 2
3 3 3 3
4 4 4 4
F90 random_number still does work not on borg, Fall 1998
function drand48() uniform random array:
0.7469419493 0.5427079092 0.0258878556 0.5654528109
0.2614139434 0.9823978669 0.8074451619 0.0462442146
0.9162790193 0.1535050702 0.7112050429 0.3834904004
0.6550487669 0.3811652680 0.6453757818 0.5827837354
0.6270208712 0.2289106442 0.7934562939 0.0215042220
0.1299298527 0.7670702197 0.4952551953 0.8667496918
%%%%%%%%%%% end f90test.output %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Compiler directives can be used to force optimization or stop optimization for both `f90', `cc' and `cpp (C Preprocessor)' compilers. These statements are placed in the Fortran or C source just before the loop or other entity they are to effect. However, some come in single statements, while others come in pairs with one beginning directives and another ending directive. The compiler directives have different formats, i.e., for `f90' they have the form:
C$DIR [directive]
while for `cc' they are called pragmas and have the form:
#pragma [_CNX] [directive]
where the old Convex marker `_CNX' is now optional. If several compatible directives are used on the same code fragment like a loop, then they may be combined, for example in the case of two directives, as
C$DIR [directive1], [directive2]
#pragma [_CNX] [directive1], [directive2]
Caution: for `f90', the leading `C' of `C$DIR' must be in column 1 and a blank must be in column 6. The `#pragma' statement keyword must be in lower case for `cc', but the directives themselves as well as `C$DIR' can be in upper or lower case. Also, it is not wise to force optimization where inappropriate and risk synchronization errors.
For HP Documentation on C Pragmas (use for Fortran Compiler Directives too, since there is very little HP Documentaion for them) see
{May not be a stable link, so you may have to try the searching the following HP documents} in
of
For more General List of HP Documentation see
HP Development Tools & Distributed Computing.
These directives effect scalar optimization or nonoptimization at the point at which the directive appears and only affects the local program unit, such as the loop it appears in.
C$DIR scalar
#pragma [_CNX] scalar {Directive prevents parallelization or data localization of the following loop. Can be used on inner loop, while larger outer loop is parallelized. Needs either +O3 or +O4. }
C$DIR no_side_effects [function_or_subroutines_list]
#pragma [_CNX] no_side_effects [procedure_list] {Allows inlining, transformations of arguments or common or input/output or other calls for named procedures, functions or subroutines. Needs either +O3 or +O4. }
C$DIR critical_section [([gate_variable])]
.......[critical_section]..........
C$DIR end_critical_section [([gate_variable])]
#pragma [_CNX] critical_section [([gate_variable])]
.......[critical_section]..........
#pragma [_CNX] end_critical_section [([gate_variable])] {Allows only a single thread (i.e., prevents parallelization) of critical (i.e., necessarily serial sections). The optional `([gate_variable])' argument is used to label a parallel task, but see the `f90' directive `C$DIR gate([gatename_list])' using `man f90' and the `typedef' in `cc'. The two directives must be used in pairs to surround the critical section. }
C$DIR no_block_loop
#pragma [_CNX] no_block_loop {Disables loop blocking or strip mining for the next loop only. Needs either +O3 or +O4. }
C$DIR block_loop[(block_factor=[number])]
#pragma [_CNX] block_loop[(block_factor=[number])] {Enables blocking or strip mining of the following loop by an optional `(block_factor=[number]' factor. Caution: directive will work only for scalar, innermost loops without conditional branching. Needs either +O3 or +O4. }
C$DIR no_loop_transform
#pragma [_CNX] no_loop_transform {Disables distribute, fuse, block, interchange, unroll, unroll and jam, or parallelize for the next loop only. Needs either +O3 or +O4. }
C$DIR no_parallel
#pragma [_CNX] no_parallel {Disables loop parallelization for the next loop only. Needs either +O3 or +O4. } o
C$DIR no_unroll_and_jam
#pragma [_CNX] no_unroll_and_jam {Disables loop unrolling and jamming (vertical unrolling or loop collapsing) for the next loop only. Needs either +O3 or +O4. }
C$DIR unroll_and_jam[(unroll_factor=[number])]
#pragma [_CNX] unroll_and_jam[(unroll_factor=[number])]
{Enables (horizontal) unrolling and jamming (vertical unrolling) of the
following non-innermost loops to an optional `unroll_factor=[number]'
unroll depth.
Caution: Directive will work only on loops that are non-innermost
after automatic loop interchanges. Needs either +O3 or +O4.
}
These compiler directives hold only for the loop immediately following the directive.
C$DIR no_loop_dependence([array_list])
#pragma [_CNX] no_loop_dependence([array_list]) {Informs compiler that arrays named in `[array_list]' do not have any dependencies for the next loop only. Needs either +O3 or +O4. See about `C$DIR loop_private' and `C$DIR thread_private' in `man f90' (or `#pragma [_CNX] loop_private' and `#pragma [_CNX] thread_private' in `man cc') about informing the compiler about scalar that have no dependencies. }
C$DIR loop_parallel[([attribute_list])]
#pragma [_CNX] loop_parallel([attribute_list]) {Enables parallelization of the next loop according to the properties in the optional `[attribute_list]' argument. Some possible attributes are `DIST' to distribute iterations over parallel threads (virtual processors to be scheduled on single physical processor upon execution); `max_threads=[number]'; number of iterations (chunks) and 'chunk_size=[number]' to be distributed to each parallel processor, with compatible attributes separated by commas. Caution: directive may not work if the number of loop iterations are not known to the compiler (e.g., computed number of iterations known only at execution) and if F90 array sections are used in implied loops. See `man f90' or `man cc' or the Exemplar Programming Guide for more information and examples. }
C$DIR parallel[([attribute_list])]
#pragma [_CNX] parallel([attribute_list]) {Marks the beginning of a parallel region of code. All code up to the following end_parallel pragma will be run on all available threads. No loop transformations, data privatization, or parallelization analysis will be performed on this region. }
C$DIR prefer_parallel
#pragma [_CNX] prefer_parallel {Informs compiler to parallelize the following loop only if it is safe to do so by checking for loop carried dependencies. }
C$DIR peel
#pragma [_CNX] peel {Enables peeling of the following loop by removing first and last iteration tests and replicating the code. }
C$DIR begin_tasks[([attribute_list])]
.......[parallel_task_first]..........
C$DIR next_task
.......[more_parallel_tasks]..........
C$DIR next_task
.......[parallel_task_last]..........
C$DIR end_tasks
#pragma [_CNX] begin_tasks[([attribute_list])]
.......[parallel_task_first]..........
#pragma [_CNX] next_task
.......[more_parallel_tasks]..........
#pragma [_CNX] next_task
.......[parallel_task_last]..........
#pragma [_CNX] end_tasks {Enables parallelization of different sections of codes (not just loops) that are separated by the `begin_tasks', perhaps several `next_task' and `end_tasks' compiler directives as indicated above for the case of two tasks `[parallel_task_first]' and `[parallel_task_last]'. HP calls this Task Parallelism in contrast to Loop Parallelism. A typical attribute is `max_threads=[number]' giving the number of threads (processors) to be used, where `threads' is the default attribute and `nodes' makes little sense at UIC. Caution: there are no dependency checks, but see information on `C$DIR task_private([variable_list])' in `man f90' or `#pragma [_CNX] task_private([variable_list])' in `man cc' to avoid unintended dependencies between tasks (note: the `begin_task' and `task_private' directives can be combined, separated by commas). }
srand48([iseed]);
[uniform]=drand48();
{Generates pseudo random numbers uniformly distributed with the value in
the `uniform_variable' argument and is part of the HP-UX
C `stdlib.h' standard libraries, but the double precision
The utility `drand48' is recommended over the simple `rand' utility.
The utilities `srand48' and `drand48' must be used in the sequence indicated.
The value of the argument seed `[iseed]' must
be nonzero to start a new random sequence.
Important uses are for Monte Carlo Random Simulations and for generating random test matrices as the following example indicates.
%%%%%%%%%%% begin drand48ctest.c %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% #include#include %%%%%%%%%%% end drand48ctest.c %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%/* Define Global Parameters */ #define m 6 #define n 4 /* Main program */ main() { int i, j; int seedval; double uniran[m][n]; double drand48(void); seedval = 123456; /* random seed srand48 and number generator drand48 MUST used together */ srand48(seedval); printf("\n function drand48() uniform random array:"); for (i = 0; i < m; ++i) { printf("\n "); for (j = 0; j < n; ++j) { uniran[i][j] = drand48(); printf("%12.8f", uniran[i][j]); } /* end j-loop */ } /* end i-loop */ } /* end main */
%%%%%%%%%%% begin drand48ctest.output %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% function drand48() uniform random array: 0.94064665 0.67025565 0.63174101 0.20673498 0.05537527 0.28843618 0.22219551 0.98991976 0.54627101 0.96289263 0.27266978 0.37790279 0.40388831 0.02924353 0.06055850 0.31290669 0.65928449 0.17063470 0.65881066 0.94268193 0.99463927 0.49473343 0.95366738 0.94078083 %%%%%%%%%%% begin drand48ctest.output %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%}
The HP performance analyzer CXperf permits analyzing the performance of both Fortran and C programs. It can be used in both X-Windows and line edit mode, but only the line edit example for Fortran code will be given here, otherwise consult the `man cxperf' command. The `CXperf' analyzer also servers as a coarse CPU timer.
f90 +pal +O3 +list -o [exec] [source].f > & [source].LIST & : The `+pal' option causes compilation of the program for performance analysis. The `+pa' option is used for optimization option level 4 with the `cc' only.
[time_variable] = etime([time_array]) : UNIX Fortran77 `etime' timer utility, whose output value is elapsed total cpu time `[time_variable]' in seconds since the job began, i.e., the sum of user cpu time in the first argument `[time_array](1)' and the system cpu time in the second argument `[time_array](2)', assuming that the time array `[time_array]' has real dimension two (2) and etime is type `real'. The utility `etime' requires `+U77' option in `f90' and `f77' commands, as do most timer utilities.. For timing large loops, `etime' overhead should be negligible; for most sizes of loops, the timing part of the code, with `!' marking comments on the statement line, might look like:
[time_variable] = dtime([time_array]) : UNIX Fortran `dtime' timer utility is like `etime' except it returns the elapsed time since the last call to `dtime', so is better for getting the timings for single loops rather than the whole program.
%%%%%%%%%%% Begin Sample ETIME Code Fragments %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
real dtime,tarray(2),ttot(100),tusr(100),tsys(100)
real tloop(100,3),tjob(3)
character*24 tchar(100)
kt = 1
tchar(kt) = `first time call'
ttot(kt) = etime(tarray) ! first time call
tusr(kt) = tarray(1)
tsys(kt) = tarray(2)
kt = kt + 1
tchar(kt) = `second call'
ttot(kt) = dtime(tarray) ! second call for overhead
tusr(kt) = tarray(1)
tsys(kt) = tarray(2)
Comment: if timer overhead is significant, correct for it.
code-continues
... more code ...
code-continues
kt = kt + 1
tchar(kt) = `begin loop 999'
ttot(kt) = dtime(tarray) ! start of loop 999
tusr(kt) = tarray(1)
tsys(kt) = tarray(2)
do 999 i = 1, N
code-continues
... rest of do ....
code-continues
999 continue
kt = kt + 1
tchar(kt) = `end loop 999'
ttot(kt) = dtime(tarray) !ttot(kt) - ttot(kt-1) = do-time
tusr(kt) = tarray(1)
tsys(kt) = tarray(2)
code-continues
... more do loops and more timing step pairs ....
code-continues
kt = kt +1
tchar(kt) = 'final total time'
ttot(kt) = dtime(tarray) !final time since last
tusr(kt) = tarray(1)
tsys(kt) = tarray(2)
tjob(3) = etime(tarray) - ttot(1) !final time
tjob(1) = tarray(1) - tusr(1)
tjob(2) = tarray(2) - tsys(1)
write(6,*) 'Loop/Step Times:'
do 99999 ks =2, kt !cpu-time for each timed loop
write(6,99998) ks, tusr(ks),tsys(ks),ttot(ks), tchar(ks)
Comment: writes hinder optimization, so save writes until last
99999 continue
write(6,*) 'Total Job Time:'
write(6,99998) kt, (tjob(j),j=1,3), tchar(kt)
99998 format(1x,i3,' (user,system,total) time =',3f11.6,
& ' seconds: ',a)
%%%%%%%%%%% End Sample ETIME Code Fragments %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
For timing small loops, put the small loop inside another loop that just does a large number of repetitions of the small loop, say N, then divide the time difference by N; use `man dtime' for other information.
[return_code] = gettimeofday([time_array],[time_zone]);
: C microsecond wall timer, where wall time is the elapse time that is the
sum of user cpu time and system cpu time.
Needs C header file `
Notes:
The best way to learn these commands is to use them in an actual computer
session.
Good luck.
FBH
Please report to Professor Hanson any problems or inaccuracies:
Web Source: http://www.math.uic.edu/~hanson/mcs572/borgguide.html
Fall 1998
%%%%%%%%%%% Begin Sample GETTIMEOFDAY C Code Fragments %%%%%%%%%%%%%%%%%%%%%%%%
/* revised, simplified use of gettimeofday */
/* Compile/Link: cc +O3 +L -lm -o [exec] [source].c >& [source].LIST & */
/* Execute: lsrun [source] >& s[source].output & */
/* Ignore EXEC MSG: "Exit 11 lsrun start >& start.output" */
#include
Table of HP9000 Timers.
HP9000 Timer Summary
Timer TimeMeasured Units Comments
----- ------------ ----- --------
ctime ConvertToText ASCII f90:converts stime
etime ElapsedUser&System Seconds f90:Microsecond Resolution
Caution: Elapsed time from job start Needs +U77 option
dtime Similar to etime, but elapsed time since last call to dtime
clock CurrentTime hh:mm:ss f90:libcfc
gettimeofday WallTime (UTC) Secs&Microsecs C;plus more;TOD;
second ElapsedWall Seconds f90:libcfc;elapsed from job start
secnds FloatWallTime Seconds f90:libU77;since midnight;
stime SysWallTime Seconds f90
timef ElapsedWall Milliseconds f90:libcfc;elapsed time since last call
times Process&Child ClockTicks cc:include