Batch / Parallel Fluent Execution

Batch / Parallel Execution of Fluent 6

Last Update: Jan 2007

The most efficient and convenient way to run Fluent solver for your CFD simulations, which need hours, days or weeks to finish, is to run the solver in LSF batch/parallel mode. First, you shall setup the CFD problem, including mesh, models, boundary conditions etc., on an interactive Fluent interface. Next save the initial problem to Fluent case and data file pair. Then you can choose a suitable batch queue to submit your case for computing.

To know how to run Batch/Parallel Fluent solver in SVU, Do read this article "Fluent Job Submission Made Easy" on Issue No. 14, HPC@NUS, January 2007

Small or medium sized jobs which may need about one hour to complete, can be submitted in serial-batch mode (running on single CPU). The serial batch queues are ia64, cpq, linux, linux64, hp, sun, sgi, sgi_large, sgi_small and sgi_night.

As an example, the following is the command used to submit a serial-batch Fluent job to queue linux:

bsub -q linux -o cpq.out "fluent 3d (or 2d) -g -i fluent_script_file"
where:

-q linux
define to sumbit to queue linux

-o linux.out
specify the filename to keep the screen output from the batch parallel job.

fluent 3d -g
run fluent 3d (3-dimensional) solver without graphics user interfaces (gui) or graphics display.

-i fluent_script_file
specify the Fluent command script file. The file contains the Fluent commands for the execution, you may use Unix text editor, such as vi, emacs or pico, to edit this file.

Here is an example of the script inside the fluent_script_file (for steady-state case) that will be executed in Fluent to read test.cas and test.dat files, and then do 1000 iterations, and finally save the results to a data file test1000.dat file and exit from Fluent environment:

rcd test.cas
it 1000
wd test1000.dat
exit
y

Take note that you need to include a "y" or "yes" at the end of script to confirm exit from Fluent. When the job is completed, you should find the final data file "test1000.dat" at the working directory where you submitted the job. To do the post-processing and visualise the results, launch a Fluent interface interactively from the command line and read the case and data files into it to do the visualisation.

Below is an example script file (bold text only) for unsteady flow simulation. The script defines to read fluent *.cas and *.dat files, set the auto-save of data file every 1000 time-step, define the time-step size as 6 (seconds) and run a total number of 10000 time steps with a max 10 iterations per time step. When the iteration finishes, save the final results to a new data file mydata.dat and exit.

/file/rcd myfile.cas
/file/autosave/data-frequency
1000
/solve/set/time-step 6
/solve/dual-time-iterate
10000 # No. of time steps#
10 # Max iterations of per time steps#
/file/wd mydata.dat
/exit
y

When the job is completed, you should find 10 (=10000/1000) sets of intermediate data file and the final data file "mydata.dat". You can use those data files to visualise the results.

Parallel Execution of Fluent 6 (Not in use now)

With the addition of the 22-CPU Compaq GS320 system, you can now get your Fluent results faster than before. The Fluent 6 application has been found to be very scalable and well parallelised.

You can run parallel jobs either on the cpq_3p (provides 3 dedicated CPUs) or the cpq_8q (provides 8 dedicated CPUs). The recommendation is to run small jobs (that takes less than 3 days to complete on single CPU) on the cpq_3p queue and larger jobs on the cpq_8p queue. You can also test run your parallel application on the cpq_8p_8gb queue (with a maximum memory usage up to 8 GB).

Here are the basic steps to set up a parallel Fluent 6 job (you can also check out the corresponding section in "User's Guide" from Fluent manuals):

Read in the .cas file with the sequential version of Fluent 6 and bring up the Grid Partition menu. Set the "Number" to 3 or 8 (depending on which batch queue you will be using). You can use the default grid partition and optimisation method or set your own preference (check out the User's Guide for details). And then click the "Partition" button to start the process. When it is done, give the initial conditions and save the .cas and .dat files and exit.
Assuming you have set up all the models/conditions/definitions properly, you can now submit the job to run on either cpq_3p or cpq_8p by:

bsub -q cpq_3p -o outputfile "mpiclean; fluent 3d (or 2d) -pvmpi -t3 -g -i fluent_script_file; mpiclean"
bsub -q cpq_8p -o outputfile "mpiclean; fluent 3d (or 2d) -pvmpi -t8 -g -i fluent_script_file; mpiclean"

where

-o outputfile
specify the filename to keep the screen output from the batch parallel job.

mpiclean
the command is executed before & after the Fluent command to clean up any MPI processes or files that may prevent the execution of next MPI job on queue cpq_3p or cpq_8p.

fluent 3d -pvmpi -t8 -g
run Fluent 3d (use 2d for 2-dimensional case) solver on 8 processors with vendor mpi library and without graphics user interfaces (gui) or graphics display.

-i fluent_script_file
specify the Fluent command script file. The file contains the Fluent commands for the execution, you may use Unix text editor, such as vi, emacs and pico, to edit this file.

Follow the similar way as described above to do the post-processing. You can skip the partition processing in Fluent as described in step 1 if you are happy with the auto partitioning done by Fluent solver using the default settings.

With the release of Linux cluster in September 2003, you can also run parallel Fluent solver at the Atlas Linux cluster. The parallel queue is linux_parallel, which supports up to a max of 16 CPUs.

The command used to submit parallel Fluent job to Linux cluster is a little bit different from that for cpq parallel queues. Here is an example:

bsub -q linux_parallel -o linux_p.out -n 4 fluent 3d (or 2d) -t0 -pnet -g -i fluent_script_file -lsf

where:

-n 4
-n 16
defines to run 4 Fluent solvers in queue linux_parallel.
defines to run 16 Fluent solvers in queue linux_parallel.

fluent 3d -t0 -pnet -g -i
run Fluent 3d (use 2d for 2-dimensional case) solvers with network mpi library and one solver on each processor, without graphics user interfaces (gui) or graphics display.

-lsf
jobs managed by LSF.

NOTE:
To ensure that the parallel fluent jobs at the queues linux and linux_parallel run properly and compete successfully, you NEED to logon to the Linux cluster (atlas0 or atlas1) to submit the batch jobs.

In June 2004, SVU has its first 64-bit HPC Linux cluster installed with the latest Itanium2 IA64 processors at that time. Users are able to run parallel Fluent simulation with the new parallel queue ia64_4p, at a max of 4 CPUs.

The command to submit parallel Fluent job to queue ia64_4p is as below:

bsub -q ia64_4p -o ia64_4p.out "fluent 3d (or 2d) -t4 -psmpi -g -i fluent_script_file"

where

-q ia64_4p
defines to run 4 Fluent solvers in queue ia64_4p.

-o ia64_4p.out specify the filename to keep the screen output from the batch parallel job.

fluent 3d -t4 -psmpi -g -i
run Fluent 3d (use 2d for 2-dimensional case) solvers on 4 processors with shared mpi library, without graphics user interfaces (gui) or graphics display.

-i fluent_script_file
Fluent job script file.

The post-processing is the same as mentioned above.

You can monitor the progress of your job with the following LSF commands:

bjobs to list all your batch jobs

bjobs -l <jobid> to list a particular batch jobs in detail

bjobs -p <jobid> to find out the pending reason of job

bpeek <jobid> to display the standard output of a particular job

bkill <jobid> to terminate a particular job

-q linux	define to sumbit to queue linux
-o linux.out	specify the filename to keep the screen output from the batch parallel job.
fluent 3d -g	run fluent 3d (3-dimensional) solver without graphics user interfaces (gui) or graphics display.
-i fluent_script_file	specify the Fluent command script file. The file contains the Fluent commands for the execution, you may use Unix text editor, such as vi, emacs or pico, to edit this file.

-o outputfile	specify the filename to keep the screen output from the batch parallel job.
mpiclean	the command is executed before & after the Fluent command to clean up any MPI processes or files that may prevent the execution of next MPI job on queue cpq_3p or cpq_8p.
fluent 3d -pvmpi -t8 -g	run Fluent 3d (use 2d for 2-dimensional case) solver on 8 processors with vendor mpi library and without graphics user interfaces (gui) or graphics display.
-i fluent_script_file	specify the Fluent command script file. The file contains the Fluent commands for the execution, you may use Unix text editor, such as vi, emacs and pico, to edit this file.

-n 4 -n 16	defines to run 4 Fluent solvers in queue linux_parallel. defines to run 16 Fluent solvers in queue linux_parallel.
fluent 3d -t0 -pnet -g -i	run Fluent 3d (use 2d for 2-dimensional case) solvers with network mpi library and one solver on each processor, without graphics user interfaces (gui) or graphics display.
-lsf	jobs managed by LSF.

-q ia64_4p	defines to run 4 Fluent solvers in queue ia64_4p.
-o ia64_4p.out	specify the filename to keep the screen output from the batch parallel job.
fluent 3d -t4 -psmpi -g -i	run Fluent 3d (use 2d for 2-dimensional case) solvers on 4 processors with shared mpi library, without graphics user interfaces (gui) or graphics display.
-i fluent_script_file	Fluent job script file.