Project On-line Deliverables: D02.2

The Parallel Environment

Installation Manual (Final)

Programme name:  ESPRIT 
Domain:  HPCN 
Project acronym:  HITERM 
Contract number:  22723 
Project title:  High-Performance Computing and Networking 
for Technological and Environmental Risk Management 
Project Deliverable: D02.2 
Related Work Package:  WP 2 
Type of Deliverable: Technical Report
Dissemination level: project internal 
Document Authors:  Peter Mieth, Steffen Unger, Matthias L. Jugel, GMD
Edited by:  Kurt Fedra, ESS 
Document Version:  1.3 (Final)
First Availability:  1997 12 28 
Last Modification:  1999 7 15 




EXECUTIVE SUMMARY

The document serves as an installation guide for basic software components such as the HITERM Simulation Server and necessary basic standard software. 

This document also describes the different parallelization strategies for the different HITERM model modules. It explains the methodology employed and gives initial results, showing the decrease in run-time and the speed-up as a function of the CPU nodes used. 

The three core parts of the model system for atmospheric releases (applies equally to the surface and groundwater components): 

  • source strength determination in a Monte Carlo frame
  • wind field computation 
  • transport calculation 

are parallelly implemented. In order to guarantee maximum flexibility and portability of the software, the PVM library was used. The programs run on a specially designed parallel machine such as MANNA (Giloi 1991, Garnatz 1996), as well as on a local workstation cluster running PVM (e.g. SUN, DEC, IBM, Silicon Graphics). 





Table of Contents

1 INTRODUCTION 

2 DOCUMENTATION OF THE HITERM SIMULATION SERVER

  • 2.1 The Simulation Server
  • 2.2 The CGI script "hiterm.pl"
  • 2.3 The Input Data Definition Files

3 PARALLEL VIRTUAL MACHINE (PVM) 

  • 3.1 What is PVM? 
  • 3.2 Unpacking 
  • 3.3 Building and Installing 
  • 3.4 Starting and Stopping PVM 

4 APPLICATION PROGRAMS 

5 MONTE CARLO SIMULATION 

6 THE DIAGNOSTIC WIND MODEL (DWM) 

7 PARALLELIZATION OF THE LAGRANGIAN DISPERSION MODEL (LPD) 

8 REFERENCES 





1 INTRODUCTION 

For emergency planning and especially for emergency management purposes, the system has to provide a result in a period of time at least an order of magnitude shorter than real time. This requires rapid data management as well as a fast execution of the underlying models. 

To date, simple screening models have been used for these purposes. These models are very fast, but very limited in their application because they normally assume a flat terrain and do not allow complex wind fields as an input; although a realistic determination of the wind field and its use in the transport model provides the basis for a reliable concentration calculation. 

Another drawback of these models is that, although the quality of the input parameters is often very questionable, they are considered as known. 

The HITERM approach uses advanced models in conjunction with uncertainty analysis methods. In general, the execution of this system in an appropriate resolution is too slow for emergency management purposes. To accelerate the execution, the numerical algorithm must be optimized with respect to the run-time requirements. Also, the use of fast hardware platforms is necessary, which implies of course, an expensive supercomputer. 

However, only a small number of customers possess or have permanent access to such a machine. To guarantee fast execution and flexible installation options, the time-consuming parts of the model code have been parallelized. An advantage of this approach lies in the flexibility of the installation. Wherever a cluster of workstations is available, irrespective of type, the program can be employed with the appropriate underlying software. The code remains scalable; depending on the actual number of workstations or CPU's the program is able to run on different configurations. 

The parallelizations use the paradigm of Parallel Virtual Machine (PVM, Geist 1994). PVM is a software package that allows the use of a heterogeneous UNIX computer cluster interconnected by a network as a single parallel computer. This guarantees a maximum of flexibility and portability to a variety of hardware platforms. PVM is available for most workstations and is available free of charge. Additionally, fine documentation, international workshops and user groups exist which facilitate the exchange of experiences made using PVM. For a customer, it is easy to install and to maintain. 

Initial run-time studies indicate that with the help of the code parallelization a considerable decrease in the run-time can be achieved. By conducting some further optimization efforts, the total execution time of a full run of the model system in the air domain (including Monte Carlo uncertainty analysis, three-dimensional wind field generation and Lagrangian transport calculation) can be reduced to 10% of real time. 
 

2 DOCUMENTATION OF THE HITERM SIMULATION SERVER

This section consists of two parts. The first part contains a description of the server scripts to start and control the simulation, and the second part describes the format of the input data definition files. The latter are necessary to generate input data files for the simulation system run by the simulation server.

2.1 The Simulation Server

The simulation server consists of a system-independent HTTP server and simulation dependent CGI scripts. The functionality of the HTTP server is that of a standard WWW server as can be found on the internet. The currently used server is a standard "Apache Server", version 1.3.1. Mandatory for the usage of the server side scripts is the ability to run CGI style programs from within the HTTP server. 

It is useful to run an HTTP server independently on its own machine, near to the actual simulation machine on its own server port. That way the server is only used for simulation requests. For security reasons it should run with its own user ID, preferably the same as the simulator user ID. An example of the server configuration is given below: 

-- httpd.conf -- 
# Server type 
ServerType stand-alone 
# Port number, where the server listens for requests 
Port 47777 
# Force a specific userid and group 
User simserv Group dymos 
# The server root directory points to the apache installation
ServerRoot /home/dymos2/simserv/bin/apache 
-- httpd.conf --

To be able to run scripts at the server side some specific configuration options have to be enabled in a second file: 

-- srm.conf -- 
# set the document root to a directory with explanations of the simulation
DocumentRoot /home/dymos2/simserv/bin/pages 
# the script alias points to the directory, where the CGI scripts # are located in this case "hiterm.pl" 
ScriptAlias /bin/ /home/dymos2/simserv/bin/models/
-- srm.conf --

As can be derived from the configuration, the installation is done using a user "simserv" in group "dymos" that has an apache installation in a directory called "/home/dymos2/simserv/bin/apache". The configuration files are located at "/home/dymos2/simserv/bin/apache/etc". The server looks for CGI scripts in the directory "/home/dymos2/simserv/bin/models" and runs those scripts as user "simserv". This is necessary to guarantee a correctly running simulation. 

2.2 The CGI script "hiterm.pl"

The CGI script is responsible for generating the initial environment for a simulation and the generation of simulation input files from the data transferred to the server by the client. The format of the data transfer is discussed earlier in this documentation and will not be discussed here. The script performs its task in three stages. Stage one is the initialization as mentioned above. Stage two prepares the input files for the simulation and in stage three the simulation is done. 

Stage 1:

To ensure that the simulation can be run, the script has to be configured to adapt it to the environment where it is run. That is done using environment variables listed in the first stage of the script: 
 

HOME 
The home directory of the simulation server.    (/home/dymos2/simserv) 
PVM_ARCH 
The PVM architecture    (SUN4SOL2, SUNcomputer, using Solaris 2)

Next the specific models used in the simulation are configured: 

A) The Diagnostic Windfield Model

DWM 
The directory where the diagnostic wind field model is run. 
 ($HOME/pvm3/bin/$PVM_ARCH) 
DWMSRC 
This is the directory where all necessary files used in the simulation are located. 
These files are copied to the working directory configurated as DWM.
($HOME/HITERM/models/dwm/dwmpar) 
DWMEXEC 
The name of the executable (the program) to be run.    (dianew)

B) The Evaporation Model in a Monte Carlo Frame

MOC 
The directory where the montecarlo evaporation model is run. 
 ($HOME/pvm3/bin/$PVM_ARCH) 
MOCSRC 
This is the directory where all necessary files used in the simulation are located. 
These files are copied to the working directory configurated as MOC. 
($HOME/HITERM/models/montecarl) 
MOCEXEC 
The name of the executable (the program) to be run. (evaprun)

C) The Lagrangian Model 

LAG 
The directory where the montecarlo evaporation model is run. 
($HOME/pvm3/bin/$PVM_ARCH) 
LAGSRC 
This is the directory where all necessary files used in the simulation are located. 
These files are copied to the working directory configurated as MOC. 
($HOME/HITERM/models/lagrange)
LAGEXEC 
The name of the executable (the program) to be run. (lagrun)

 

 
 
 

All three models are parallelized and run in the PVM executable directory. The output of the simulation run is located in a file configured with the variable OUT, which is set to "$ENV{LAG}/outess.dat", the output of the last of the three models used. The script then checks the existence of the directories and will create those directories that are missing. To ensure that only one simulation is run at a time the script locks the simulation server and returns an error if a simulation is in progress. 

Stage 2:

Stage two is responsible for creating the simulation input files. This is done by parsing the transferred client data and writing the files according to their format definition discussed below. The script waits for input data up to 60 seconds the first time and up to 10 seconds for each successive line. 

Stage 3:

In stage three a shell script is called which handles the run of the models, and the output of all activities is collected. If an error occurred the script will generate an error message and abort. The same happens if simulation output is not produced, which indicates an error during the simulation. 
In the case that the simulation did run and produced output, that output is transferred back to the client (please note that the connection will be maintained during the whole simulation) and the simulation server is unlocked for further requests. 

The "hiterm_run.sh" script is used to start all simulation programs. The script first copies all executables and data files to the working directories and then checks the existence of all necessary input data files. If these preconditions are met the script first initializes the PVM and then starts the diagnostic wind field model and the Monte Carlo evaporation model simultaneously. Both models have to finish working before the Lagrangian simulation is started. Before exiting, the "hiterm_run.sh" script shuts down the parallel virtual machine to clean the system up. 

2.3 The Input Data Definition Files

To make the input data files configurable and independent of the simulation server program a data definition file is used that describes for each of the keywords sent by the client how it corresponds to the line in the input file and how to format the data. The input data definition file defines for each line of the input file which keyword data has to be replaced at that position. The format of such a definition file is as follows: 
 
keyword format replacement description

The keyword has to be defined in the transfer protocol and the data accompanying the keyword will be formatted as defined in the format, which is similar to the printf() format in C. In case the data for the keyword is missing, it may be replaced by the keyword given in the replacement column. If the `-' sign is present in the replacement column, that data entry is mandatory. However, it may be left out if the `?' sign is present. As an example the input data definition file for meteorological data for the Monte Carlo frame of the evaporation model is shown below: 

-- momet.def -- 
# MONTE CARLO EVAPORATION MODEL - meteorological data definition file #
 
# keyword
MIN_TOTAL_VOL
TOTAL_VOL
MAX_TOTAL_VOL
MIN_SPILL_DURATION
SPILL_DURATION
MAX_SPILL_DURATION
MIN_DEPTH
DEPTH
MAX_DEPTH
MIN_CLOUD
CLOUD
MAX_CLOUD
MIN_WIND_SPEED
WIND_SPEED
MAX_WIND_SPEED
MIN_SURFACE_TEMP
SURFACE_TEMP
MAX_SURFACE_TEMP
format
%-20.20s%s
%-20.20s%s
%-20.20s%s
%-20.20s%s
%-20.20s%s
%-20.20s%s
%-20.20s%s
%-20.20s%s
%-20.20s%s
%-20.20s%s
%-20.20s%s
%-20.20s%s
%-20.20s%s
%-20.20s%s
%-20.20s%s
%-20.20s%s
%-20.20s%s
%-20.20s%s
replace
TOTAL_VOL
-
TOTAL_VOL
SPILL_DURATION
-
SPILL_DURATION
DEPTH
-
DEPTH
CLOUD
-
CLOUD
WIND_SPEED
-
WIND_SPEED
SURFACE_TEMP
-
SURFACE_TEMP
description
min
total vol (m3)
max
min
spill duration(s)
max
min
depth (m)
max
min
cloud cover (%)
max
min
wind speed
max
min
surface temp (K)
max
-- momet.def --

All data entries are formatted using 20 characters left adjusted for the description of the data entry and the data value is appended to the line as transferred to the server by the client. The description part is for explanation only and does not have any influence on the input data. If the description column contains a `-' sign, it indicates an empty string as in our following example for the input data definition file of the diagnostic wind field model: 

-- dwminput.def -- 
# DIAGNOSTIC WIND FIELD MODEL - input definition file # 
 
# keyword
REGION
START_TIME
NX
NY
DX
DY
GAMMA
STABILITY_CLASS
MIXING_HEIGHT
WIND_SPEED
WIND_DIRECTION
MEASUREM_HEIGHT
ROUGHNESS_LENGTH
SURFACE_TEMP
format
%s%s
%-20.20s%s
%-20.20s%s
%-20.20s%s
%-20.20s%s
%-20.20s%s
%-20.20s%s
%-20.20s%s
%-20.20s%s
%-20.20s%s
%-20.20s%s
%-20.20s%s
%-20.20s%s
%-20.20s%s
replace
-
-
-
-
-
DX
-
-
-
-
-
-
-
-
description
-
starting time
nx
ny
dx (M)
dy (m)
gamma k/m
stability class
mixing height
wind speed
wind direction
meas. height
roughness length
surface temp (K)
-- dwminput.de --

All lines containing a `#' as their first character will be ignored by the script. 

3 Parallel Virtual Machine (PVM)

PVM has been chosen in order to ensure a maximum level of portability and to guarantee an easy installation. The design of the message passing programming model PVM is shown in Figure 1. (Design of PVM [Geist 1994]) 

A brief introduction on how to get and how to install PVM has been published by Beguelin et al. (Beguelin 1992) from which Section 3.1 to 3.4 of this document quote. 

Fig. 1: Design of PVM  [Geist 1994]

3.1 What is PVM?

PVM is a software system which enables a collection of heterogeneous computers to be used as a coherent and flexible concurrent computational resource. The individual computers may be shared or local-memory multiprocessors, vector supercomputers, specialized graphics engines, or scalar workstations, which may be interconnected by a variety of networks, such as ethernet, or FDDI. User programs written in C, C++ or Fortran access PVM through library routines. 

3.2 Unpacking

This distribution contains source code, simple examples, and run-time support for PVM version 3. The documentation for PVM can be obtained separately from Netlib. To request a list of available subsets, send an e-mail to "netlib@ORNL.GOV" with the subject: "send index from pvm3". Files in the distribution unpack in directory pvm3. The pvm3 directory can reside in either a private or shared disk area. 

Installations for multiple machine architectures can coexist because compiled files are placed in subdirectories named for each architecture (ARCH). Some of the more important directories are: 
Directory Contains
bin/ARCH  PVM user program executables (the examples or your programs). 
conf  Make configuration files for all PVM architectures. 
console  Source for the pvm console. 
doc  Miscellaneous documentation. 
examples  Example PVM program source. 
gexamples  More example PVM programs - for group library. 
hoster  An example "hoster" program. 
include  Header files for PVM programs. 
lib  Generic system executables (scripts). 
lib/ARCH  System executables (pvmd, console, etc.). 
libfpvm  Source for the libfpvm Fortran library. 
man/man[13] Online manual pages (nroff format). 
misc  Some PVM examples and utilities. 
patches  Patch files and instructions, as they are released. 
pvmgs  Source for the libgpvm library and group nameserver. 
src  Source for the libpvm library and pvmd daemon. 
src/ARCH Additional source code for specific machines. 
tasker  An example "tasker" program. 
xep  An example interactive X-Window program. 

3.3 Building and Installing

Before building or running PVM, you must set environment variable PVM_ROOT to the path where PVM resides, i.e. the path of this directory. This can be in a private area, for example $HOME/pvm3, or a public one, such as /usr/local/pvm3

If your shell is csh, add a line such as: 
setenv PVM_ROOT $HOME/pvm3
to your .cshrc file. If you use a shell that reads .profile, such as sh or ksh, add the following lines to that file: 

PVM_ROOT=$HOME/pvm3
PVM_DPATH=$PVM_ROOT/lib/pvmd
export PVM_ROOT PVM_DPATH
The use of these variables is explained more fully in the pvm_intro man page. Type "make" in this directory. Make will use aimk to build the daemon executable (pvmd3), C library (libpvm3.a), Fortran library (libfpvm3.a) and console client program (pvm). 

The libraries and executables are installed in $PVM_ROOT/lib/ARCH/, where ARCH is the host architecture name, e.g. "SUN4". 

The scripts $PVM_ROOT/lib/pvm and $PVM_ROOT/lib/pvmd are used, respectively, to start the PVM console and pvmd. They determine the machine architecture and run the actual programs in $PVM_ROOT/lib/ARCH. You can either copy these scripts to your bin directory or add $PVM_ROOT/lib to your shell search path. 

You may wish to add $PVM_ROOT/man to your MANPATH environment variable, if it is supported on your system. This will allow you to easily read the online manual pages. 

3.4 Starting and Stopping PVM

To start PVM, run $PVM_ROOT/lib/pvm. This starts the console task, which in turn starts a pvmd if one is not already running. More hosts can be started by using the console "add" command. To stop PVM, use the PVM console command "halt". From programs, use the pvm_halt() function. You can also kill the pvmd (always use a catchable signal). For more information about the console commands, see the console "help" function or console man page. To start the pvmd without starting the console, run pvm3/lib/pvmd. A number of hosts can be started by supplying the pvmd with a host file. 

4 APPLICATION PROGRAMS 

C, C++ and Fortran programs must be linked with pvm3/lib/ARCH/libpvm3.a. Fortran programs must also be linked with pvm3/lib/ARCH/libfpvm3.a. Include C/C++ header file pvm3/include/pvm3.h for constants and function prototypes. Fortran header file is in pvm3/include/fpvm3.h. Executables should be installed in pvm3/bin/ARCH. This path can be changed; see the pvmd man page. (paragraph quoted from [Beguelin 1992]) 

Application users do not have to know a lot about PVM. Once the software is installed correctly, it is mainly some environment variables which have to be set. 

The computational part of HITERM consists of several shell scripts which: 

  • interpret the data stream coming from the server 

  • start pvm (if necessary) 

  • execute the calculation tasks 

  • deliver the results to the server. 

To install and use this part, the user has to do the following: 
  • ensure that PVM software is installed properly (private or public) and that the environment variable PVM_ROOT is set; set PVM_ARCH to the current architecture (e.q. by adding the following to .cshrc) 


  • setenv PVM_ARCH `$PVM_ROOT/lib/pvmgetarch`
  • install on the home directory directories pvm3/bin/PVM_ARCH for each architecture you want to use 

  • define the file hosts in directory pvm3, containing the names of the platforms you want to use, e.g. 

  • platform1
    platform2
    platformn

You should have login access to these machines without password (compare your .rhosts file). The number of workstations in this file must not match the number of processes generated by the pvm programs themselves, possibly not all workstations will be used or several processes will be mapped to one machine. 

5 Monte Carlo Simulation

Source strength models for emergency management tasks suffer from a high degree of uncertainty regarding the input parameter set. Depending on the sensitivity of the source function, this often causes unrealistic concentration patterns. To allow a more realistic determination of the release, a Monte Carlo method is used to construct a probability function of the source strength for different times of the release duration. With this method, all uncertain parameters are varied in a selected range. The parallel implementation of the source term models uses a generalized mask to run different types of source term models. 

The user must specify the number of input parameters which are not precisely known. In addition, an uncertainty range must be given for every parameter. This range can vary in positive or negative direction. 

A probability function of the source strength is constructed using hundreds of thousands of runs of a deterministic release model. The input parameters of this model are varied over the given uncertainty range using a random generator. The parallelization of this Monte Carlo process simply exploits the complete independence of each parameter run. So the same piece of code runs for the total time sequence on the selected number of nodes (or workstations). Since random number generators do not produce random numbers but pseudo random numbers, it has been ensured and tested that every node computes a different sequence of random numbers which show a uniform distribution. 

In the initialization phase of the program, the master starts with a large but reasonable number for defining the first-guess source strength distribution (currently set to 1000). A predefined rule system for the program truncation helps to check whether a nearly smooth distribution has been reached. If not, more runs are executed since the truncation rules are valid. The truncation rules take into account the derivation of statistical characteristics of the distribution (e.g. the mean of the distribution) between two steps. If it exceeds a given epsilon (reasonable small number), another set of simulations start. Otherwise, it proceeds to an additional set of variations. 

6 The Diagnostic Wind Model

One of the main features of the DWM (Douglas 1990) is the solution of the continuity equation. In its simplified formulation, a divergence minimization procedure ensures the construction of a mass-consistent wind field. 

The first divergence parametrization is executed after the parametrization of the kinematic effects. At the end of further parametrization steps, the vertical velocity is adjusted to avoid unrealistically high vertical components at the top of the model domain. This again creates divergences in the wind, and for this reason the divergence minimization procedure is executed twice. 

The divergence minimization is an iterative process using a predefined tolerance (a reasonably small number in the range of 0.0001). This process can easily account for more than 90 % of the total CPU time of the DWM. 

In the original version of the code, a computation of a domain of 300 x 300 x 20 grid points costs several hours of execution time on a fast workstation (SUN Ultra)! Redesigning the code and optimizing the responsible loops decreased the execution time considerably. Nevertheless, the model was still too slow to fulfill the user requirement for a much faster than real time execution. 

Analyzing the code, it became obvious that the parallelization must exploit the independence of the divergence computation in every vertical level. The different vertical levels are distributed step-wise among the processors, beginning from the bottom. Usually, the layers close to the surface show the largest divergences, thus requiring many more iteration steps before obtaining the accepted tolerance criteria, as represented by the levels at the top of the domain. 

This results in a strong load balance problem if the number of nodes is not appropriate in relation to the number of vertical levels used. In the original version of the DWM, the number of vertical levels was an input parameter together with the vertical resolution between the individual vertical layers. For emergency purposes, the system has to avoid all non-physical and/or inappropriate choices by a customer. For this reason, the user can no longer control the number of vertical levels used in the model. 

The number varies between 14 (for mixing heights smaller than 500 m) and 20 for all other cases. This also simplifies the control of an effective distribution of the vertical levels among the processors. 


 

Fig. 2: Execution time versus number of processors for the diagnostic wind model (run time studies)

Initial results of the run-time characteristics of the parallel program for different number of nodes are given in Figure 2. Figure 2 was computed on a 300 x 300 x 14 grid domain using a workstation-cluster of SUN U2/2170Cr 2 x Ultra-SPARC-I 167 MHz workstations with 2 processors each, connected by an Ethernet and running PVM3 under Solaris 2.5.1. It is clearly visible that the execution time cannot be reduced with more than 7 processors. The reason can be found in the load balancing problems if the number of vertical levels is too small in relation to the number of nodes. The optimum configuration consists of 5 processors. The speed-up is rather high, reaching a value of approximately 4.2 with a total execution time in the range of 1 minute which meets the user requirements. 

7 Parallelization of the Lagrangian Dispersion Model (LPD) 

In the Lagrangian approach of the dispersion calculation, concentrations of gases are represented by particles as well as real particulate matter. The model uses winds and fluctuation computed by the DWM to predict the pathways of individual particles or air volumes. As long as an output time step has not been reached, the individual particles can be considered as fully independent. This makes the model well suited for numerical parallelization. 

The amount of emitted particles per time step representing a certain released mass is usually chosen to an acceptable but relatively small number. Unfortunately, there is no theory to determine the appropriate number of particles for the desired application. That is why the selection of this number is often left to choice, or the number is limited by the hardware resources. 

Insufficient particle numbers lead to unrealistic results, bad resolution and rippled concentration patterns. Test runs cannot be generalized and can only serve as a first approach. In order to ensure that calculations are, in most cases, on the right side, the use of somewhat large particle ensembles is recommended derived from test runs with different particle numbers. However, neglecting the time required for an output (and computing the related concentration patterns), the execution time is nearly linear proportional to the number of transported particles. A parallel implementation offers the possibility of using a large set of particles and guarantees a reasonable execution time. 

Fig. 3: Execution time versus number of processors for the Lagrangian particle model (run time studies)
 
 

Figure 3 shows initial results of run-time studies with a Lagrangian particle model using the following parameters, hardware and software environments: 

  • total duration was 300 time steps of 10 seconds 

  • 200 particles were emitted per time step (leads to 60 000 in the end) 

  • the particles were equally distributed among processors - i.e., for 8 processors 25 particles were released per time step 

  • Parallel Virtual Machine (PVM 3) 

  • hardware: 4 workstation-clusters of SUN 20/712TGX, 2x SuperSPARC-II 75 MHz workstations with 2 processors each, Ethernet connection, SUNOS 4.1.3_U1, 128 Mb Memory. 

The results indicate an ideal speed-up of 8 for 8 processors and a reduction of the total execution time from approximately 1085 seconds for the single processor run to 136 seconds for the cluster of 8 processors. Additionally, the problem remains scalable which offers the possibility of enlarging the number of processors considerably, with an expected further reduction of the execution time. 

Bottleneck efficiency can be found in the final output computation. As soon as a concentration calculation is required (because an output is desired), the particles are no longer independent and communication becomes necessary. Therefore, the number of output intervals should be reasonably small in order to avoid excessive decrease of the speed-up of the parallel implementation. 

8 References

Beguelin A.L., Dongarra J. J., Geist A., Jiang W. C., Manchek R. J., Moore B. K. and Sunderam V. S. (1992) 
PVM version 3.3: Parallel Virtual Machine System. University of Tennessee, Knoxville, TN, Oak Ridge National Laboratory, Oak Ridge, TN, Emory University, Atlanta, GA. 
Douglas G.S., Kessler R.C., Carr L. (1990) 
User's Manual for the Diagnostic Wind Model. EPA-450/4-90-007C. 
Garnatz Th., Haack U., Sander M., Schröder-Preikschat W. (1996) 
Experiences made with the Design and Development of a Message-Passing Kernel for a Dual-Processor-Node Parallel Computer. In Proceedings of the Twenty-Ninth Annual Hawaii International Conference on System Sciences. (Maui, Hawaii, January 3-6, 1996). IEEE Computer Society Press. 
Geist A., Beguelin A., Dongarra J., Jiang W., Manchek R., Sunderam V. (1994) 
PVM: Parallel Virtual Machine, A User's Guide and Tutorial for Networked Parallel Computing. The MIT Press, Cambridge, Massachusetts 
Geist A. (1994) 
EuroPVM. MPI'97, Krakow, Poland. 
Gerharz I., Lux Th., Sydow A. (1997) 
Inclusion of Lagrangian Models in the DYMOS System. In Proceedings of the IMACS World Congress 1997. (Berlin, Germany, August 25-29). W&T, Berlin, 6: 53-58. 
Giloi W.K., Brüning U. (1991) 
Architectural Trends in Parallel Supercomputers. In Proceedings of the Second NEC International Symposium on Systems and Computer Architectures. (Tokyo, Japan, August). Nippon Electric Corp. 

Copyright 1995-2002 by:   ESS   Environmental Software and Services GmbH AUSTRIA