Submitting COMSOL jobs to HPC

seekzzh included in Practical Skills

2021-11-25 1591 words 8 minutes - views

Contents

Basic workflow

Here, we use the tensile test of a solid cube as an example.

Export .m file from COMSOL.

In COMSOL: In the File Menu, Click Compact History, then Save as the .m file.

Modify the .m file.

At first, you need to the code about modelPath and label. These two codes are generated due to exportation from COMSOL, and don’t make sense in HPC cluster.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


function out = model
%

% <Code block 1 to be added>

% Cube_tension.m
import com.comsol.model.*
import com.comsol.model.util.*

model = ModelUtil.create('Model');

% <Code to be deleted>
model.modelPath('C:\Users\Dell\Desktop');
model.label('cube_tension.mph');
% </Code to be deleted>

model.param.set('F_z', '10000[N]');
model.param.set('F_x', '0[N]');
...
...
...
model.nodeGroup('dset1solidlgrp').label('Applied Loads (solid)');
model.nodeGroup('dset1solidlgrp').add('plotgroup', 'pg2');

% <Code block 2 to be added>

out = model;

Then add the following code to the position of Code block 1.

1
2
3
4
5


% % Add the path to COMSOL
addpath('/apps/generic/software/COMSOL/5.5/mli');
% 
% % Connect to the COMSOL server on port 2036
mphstart(2036);

After that, add the following code to the position of Code block 2. This is to give a name to your final result file.

1

mphsave(model,'<name>');

Created bash(.sh) file

Here, we can create a bash file named MatCom.sh.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


#!/bin/bash
#SBATCH --time=<0-10:00:00>
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=<24>
#SBATCH --job-name=<cube_tension>
#SBATCH --output=<cube_tension>.out
#SBATCH --mem-per-cpu=<2G>  or #SBATCH --mem=<20G>
#SBATCH --partition=<regular>

#SBATCH --error=slurm-%j.stderr
#SBATCH --mail-type=ALL
#SBATCH --mail-user=<youremail@rug.nl>

module purge

module load MATLAB/2019b

module load COMSOL/5.5

# Start the COMSOL server, usually on port 2036, if it is available, otherwise it will generate an error
comsol mphserver -np ${SLURM_CPUS_PER_TASK} -port 2036 -silent -tmpdir $TMPDIR &

# Run a MATLAB script called "test.m" which uses some COMSOL functions
matlab -nodisplay -r <cube_tension>

Here, the < and > do not exist in real codes, which are just used to mark the parameters to be modified accordingly.

--time The time should be roughly estimated, and better longer than the estimated time. Otherwise, the job will be killed when the pre-defined time runs out.
--mem There are two methods to set the estimated memory.
--partition There are several ports to select.
--mail You can write your email here, and you will get infored when your job starts and ends.
job-name The job-name should the same as the --output, matlab -nodisplay -r ....

Start, monitor and cancel the job

Before starting, you need to upload the .m and .sh files to your project folder. Here, if your result file will exceed 25GB, you’d better to upload files to the /data folder that is of 250GB capability.

Submit the job, in the same folder
sbatch MatCom.sh
Check the status
squeue -u p123456
Cancel on specific job
scancel <jobid>
Cancel multiple jobs at once scancel --state=PENDING --partition=<short>

More information

The status of a job

PD Pending, the job is waiting
R Running, the job is running on one or more nodes of the cluster
CG Job is completing

Explanation of squeue output

JOBID The job id, which is used by the system to refer to the job
PARTITION Tbe partition the job is submitted in
USER The user id of the user that submitted the job
NAME The name that has been given to the job by the user
ST The job status. This will be explained below
TIME The time the job is running
NODES The number of nodes requested by the job. The number of cores requested on these nodes is not shown
NODELIST(REASON) The reason a job is waiting (explained below) or the nodes allocated to the job.

The reasons for not having started yet can be the following.

(Resources) The job is waiting for resources to be available
(Priority) The job does not have enough priority compared to other jobs
(ReqNodeNotAvail) The nodes required for the job are not available. This can be because of upcoming maintenance, or nodes that are down because of issues.
(QosGrpCpuLimit) The job has hit the limits on the number of cores that are allowed to be in use for long running jobs.

PS:

The job cannot run with license, when COMSOL is used on the work PC/UWP.
In general, the COMSOL cannot be used at the same time on Cluster/ work PC/UWP with your User name.

Schedule: partitions

Compute nodes

Storage

Appendix Example

cube_tension.m

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189


function out = model
%
% % Add the path to COMSOL
addpath('/apps/generic/software/COMSOL/5.5/mli');
% 
% % Connect to the COMSOL server on port 2036
mphstart(2036);

%
% Cube_tension.m
%

import com.comsol.model.*
import com.comsol.model.util.*

model = ModelUtil.create('Model');

model.param.set('F_z', '10000[N]');
model.param.set('F_x', '0[N]');
model.param.set('A', '0.01[m^2]');
model.param.set('L_0', '0.1[m]');
model.param.set('E', '200e9[Pa]');

model.component.create('comp1', true);

model.component('comp1').geom.create('geom1', 3);

model.result.table.create('tbl1', 'Table');
model.result.table.create('tbl2', 'Table');

model.component('comp1').mesh.create('mesh1');

model.component('comp1').geom('geom1').geomRep('comsol');
model.component('comp1').geom('geom1').create('blk1', 'Block');
model.component('comp1').geom('geom1').feature('blk1').set('size', [0.1 0.1 0.1]);
model.component('comp1').geom('geom1').run;

model.component('comp1').material.create('mat2', 'Common');
model.component('comp1').material('mat2').propertyGroup.create('Enu', 'Young''s modulus and Poisson''s ratio');

model.component('comp1').physics.create('solid', 'SolidMechanics', 'geom1');
model.component('comp1').physics('solid').create('fix1', 'Fixed', 2);
model.component('comp1').physics('solid').feature('fix1').selection.set([3]);
model.component('comp1').physics('solid').create('pc1', 'PeriodicCondition', 2);
model.component('comp1').physics('solid').feature('pc1').selection.set([1 6]);
model.component('comp1').physics('solid').create('pc2', 'PeriodicCondition', 2);
model.component('comp1').physics('solid').feature('pc2').selection.set([2 5]);
model.component('comp1').physics('solid').create('bndl1', 'BoundaryLoad', 2);
model.component('comp1').physics('solid').feature('bndl1').selection.set([4]);

model.component('comp1').mesh('mesh1').create('auto_f1', 'FreeTet');

model.result.table('tbl1').comments('Poisson Ratio');
model.result.table('tbl2').comments('Young''s Modulus');

model.component('comp1').material('mat2').label('Iron');
model.component('comp1').material('mat2').set('family', 'iron');
model.component('comp1').material('mat2').propertyGroup('def').set('relpermeability', {'4000' '0' '0' '0' '4000' '0' '0' '0' '4000'});
model.component('comp1').material('mat2').propertyGroup('def').descr('relpermeability_symmetry', '');
model.component('comp1').material('mat2').propertyGroup('def').set('electricconductivity', {'1.12e7[S/m]' '0' '0' '0' '1.12e7[S/m]' '0' '0' '0' '1.12e7[S/m]'});
model.component('comp1').material('mat2').propertyGroup('def').descr('electricconductivity_symmetry', '');
model.component('comp1').material('mat2').propertyGroup('def').set('thermalexpansioncoefficient', {'12.2e-6[1/K]' '0' '0' '0' '12.2e-6[1/K]' '0' '0' '0' '12.2e-6[1/K]'});
model.component('comp1').material('mat2').propertyGroup('def').descr('thermalexpansioncoefficient_symmetry', '');
model.component('comp1').material('mat2').propertyGroup('def').set('heatcapacity', '440[J/(kg*K)]');
model.component('comp1').material('mat2').propertyGroup('def').descr('heatcapacity_symmetry', '');
model.component('comp1').material('mat2').propertyGroup('def').set('relpermittivity', {'1' '0' '0' '0' '1' '0' '0' '0' '1'});
model.component('comp1').material('mat2').propertyGroup('def').descr('relpermittivity_symmetry', '');
model.component('comp1').material('mat2').propertyGroup('def').set('density', '7870[kg/m^3]');
model.component('comp1').material('mat2').propertyGroup('def').descr('density_symmetry', '');
model.component('comp1').material('mat2').propertyGroup('def').set('thermalconductivity', {'76.2[W/(m*K)]' '0' '0' '0' '76.2[W/(m*K)]' '0' '0' '0' '76.2[W/(m*K)]'});
model.component('comp1').material('mat2').propertyGroup('def').descr('thermalconductivity_symmetry', '');
model.component('comp1').material('mat2').propertyGroup('Enu').set('youngsmodulus', '200e9[Pa]');
model.component('comp1').material('mat2').propertyGroup('Enu').descr('youngsmodulus_symmetry', '');
model.component('comp1').material('mat2').propertyGroup('Enu').set('poissonsratio', '0.29');
model.component('comp1').material('mat2').propertyGroup('Enu').descr('poissonsratio_symmetry', '');

model.component('comp1').physics('solid').feature('pc1').set('PeriodicType', 'userdef');
model.component('comp1').physics('solid').feature('pc1').set('PeriodicIn', [1; 0; 0]);
model.component('comp1').physics('solid').feature('pc1').set('ComponentPeriodicType', {'AntiPeriodicity'; 'Continuity'; 'Continuity'});
model.component('comp1').physics('solid').feature('pc2').set('PeriodicType', 'userdef');
model.component('comp1').physics('solid').feature('pc2').set('PeriodicIn', [0; 1; 0]);
model.component('comp1').physics('solid').feature('pc2').set('ComponentPeriodicType', {'AntiPeriodicity'; 'AntiPeriodicity'; 'Continuity'});
model.component('comp1').physics('solid').feature('bndl1').set('LoadType', 'TotalForce');
model.component('comp1').physics('solid').feature('bndl1').set('Ftot', {'F_x'; '0'; 'F_z'});
model.component('comp1').physics('solid').feature('bndl1').set('FperArea', {'0'; '0'; 'F_z'});

model.component('comp1').mesh('mesh1').feature('size').set('hauto', 4);
model.component('comp1').mesh('mesh1').run;

model.study.create('std1');
model.study('std1').create('stat', 'Stationary');

model.sol.create('sol1');
model.sol('sol1').study('std1');
model.sol('sol1').attach('std1');
model.sol('sol1').create('st1', 'StudyStep');
model.sol('sol1').create('v1', 'Variables');
model.sol('sol1').create('s1', 'Stationary');
model.sol('sol1').feature('s1').create('fc1', 'FullyCoupled');
model.sol('sol1').feature('s1').create('d1', 'Direct');
model.sol('sol1').feature('s1').create('i1', 'Iterative');
model.sol('sol1').feature('s1').feature('i1').create('mg1', 'Multigrid');
model.sol('sol1').feature('s1').feature('i1').feature('mg1').feature('pr').create('so1', 'SOR');
model.sol('sol1').feature('s1').feature('i1').feature('mg1').feature('po').create('so1', 'SOR');
model.sol('sol1').feature('s1').feature.remove('fcDef');

model.result.numerical.create('av5', 'AvSurface');
model.result.numerical.create('av6', 'AvSurface');
model.result.numerical('av5').selection.set([4]);
model.result.numerical('av5').set('probetag', 'none');
model.result.numerical('av6').selection.set([4]);
model.result.numerical('av6').set('probetag', 'none');
model.result.create('pg1', 'PlotGroup3D');
model.result.create('pg2', 'PlotGroup3D');
model.result('pg1').create('surf1', 'Surface');
model.result('pg1').feature('surf1').create('def', 'Deform');
model.result('pg2').create('arws1', 'ArrowSurface');
model.result('pg2').create('surf1', 'Surface');
model.result('pg2').feature('arws1').create('col', 'Color');
model.result('pg2').feature('arws1').create('def', 'Deform');
model.result('pg2').feature('arws1').feature('col').set('expr', 'comp1.solid.bndl1.F_A_Mag');
model.result('pg2').feature('surf1').set('expr', '1');
model.result('pg2').feature('surf1').create('def', 'Deform');

model.nodeGroup.create('dset1solidlgrp', 'Results');
model.nodeGroup('dset1solidlgrp').set('type', 'plotgroup');
model.nodeGroup('dset1solidlgrp').placeAfter('plotgroup', 'pg1');

model.sol('sol1').attach('std1');
model.sol('sol1').feature('s1').feature('aDef').set('cachepattern', true);
model.sol('sol1').feature('s1').feature('fc1').set('linsolver', 'd1');
model.sol('sol1').feature('s1').feature('d1').label('Suggested Direct Solver (solid)');
model.sol('sol1').feature('s1').feature('i1').label('Suggested Iterative Solver (solid)');
model.sol('sol1').feature('s1').feature('i1').set('nlinnormuse', true);
model.sol('sol1').feature('s1').feature('i1').feature('mg1').feature('pr').feature('so1').set('relax', 0.8);
model.sol('sol1').feature('s1').feature('i1').feature('mg1').feature('po').feature('so1').set('relax', 0.8);
model.sol('sol1').runAll;

model.result.numerical('av5').label('Young''s Modulus');
model.result.numerical('av5').set('table', 'tbl2');
model.result.numerical('av5').set('expr', {'(F_z*L_0)/(A*w)'});
model.result.numerical('av5').set('unit', {'N/m^2'});
model.result.numerical('av5').set('descr', {'E'});
model.result.numerical('av5').set('const', {'solid.refpntx' '0' 'Reference point for moment computation, x coordinate'; 'solid.refpnty' '0' 'Reference point for moment computation, y coordinate'; 'solid.refpntz' '0' 'Reference point for moment computation, z coordinate'});
model.result.numerical('av6').label('Poisson Ratio');
model.result.numerical('av6').set('table', 'tbl1');
model.result.numerical('av6').set('expr', {'F_x/A' 'u/L_0' '(F_x/A)/(u/L_0)' '((E)/(2*((F_x/A)/(u/L_0))))-1' 'u'});
model.result.numerical('av6').set('unit', {'N/m^2' '1' 'N/m^2' '1' 'm'});
model.result.numerical('av6').set('descr', {'Shear Stress' 'Shear Strain' 'Shear Modulus' 'Poisson Ratio' 'Displacement field, X component'});
model.result.numerical('av6').set('const', {'solid.refpntx' '0' 'Reference point for moment computation, x coordinate'; 'solid.refpnty' '0' 'Reference point for moment computation, y coordinate'; 'solid.refpntz' '0' 'Reference point for moment computation, z coordinate'});
model.result.numerical('av5').setResult;
model.result.numerical('av6').setResult;
model.result('pg1').label('Stress (solid)');
model.result('pg1').feature('surf1').set('const', {'solid.refpntx' '0' 'Reference point for moment computation, x coordinate'; 'solid.refpnty' '0' 'Reference point for moment computation, y coordinate'; 'solid.refpntz' '0' 'Reference point for moment computation, z coordinate'});
model.result('pg1').feature('surf1').set('colortable', 'RainbowLight');
model.result('pg1').feature('surf1').set('resolution', 'normal');
model.result('pg1').feature('surf1').feature('def').set('scale', 19944.10413027234);
model.result('pg1').feature('surf1').feature('def').set('scaleactive', false);
model.result('pg2').label('Boundary Loads (solid)');
model.result('pg2').set('titletype', 'custom');
model.result('pg2').set('typeintitle', false);
model.result('pg2').set('descriptionintitle', false);
model.result('pg2').set('unitintitle', false);
model.result('pg2').set('frametype', 'spatial');
model.result('pg2').set('showlegendsunit', true);
model.result('pg2').feature('arws1').label('Boundary Load 1');
model.result('pg2').feature('arws1').set('expr', {'solid.bndl1.F_Ax' 'solid.bndl1.F_Ay' 'solid.bndl1.F_Az'});
model.result('pg2').feature('arws1').set('descr', 'Load (spatial frame)');
model.result('pg2').feature('arws1').set('const', {'solid.refpntx' '0' 'Reference point for moment computation, x coordinate'; 'solid.refpnty' '0' 'Reference point for moment computation, y coordinate'; 'solid.refpntz' '0' 'Reference point for moment computation, z coordinate'});
model.result('pg2').feature('arws1').set('placement', 'gausspoints');
model.result('pg2').feature('arws1').feature('col').set('coloring', 'gradient');
model.result('pg2').feature('arws1').feature('col').set('topcolor', 'red');
model.result('pg2').feature('arws1').feature('def').set('scale', 0);
model.result('pg2').feature('arws1').feature('def').set('scaleactive', true);
model.result('pg2').feature('surf1').active(false);
model.result('pg2').feature('surf1').label('Gray Surfaces');
model.result('pg2').feature('surf1').set('const', {'solid.refpntx' '0' 'Reference point for moment computation, x coordinate'; 'solid.refpnty' '0' 'Reference point for moment computation, y coordinate'; 'solid.refpntz' '0' 'Reference point for moment computation, z coordinate'});
model.result('pg2').feature('surf1').set('coloring', 'uniform');
model.result('pg2').feature('surf1').set('color', 'gray');
model.result('pg2').feature('surf1').set('resolution', 'normal');
model.result('pg2').feature('surf1').feature('def').set('scale', 0);
model.result('pg2').feature('surf1').feature('def').set('scaleactive', true);

model.nodeGroup('dset1solidlgrp').label('Applied Loads (solid)');
model.nodeGroup('dset1solidlgrp').add('plotgroup', 'pg2');

mphsave(model,'cube_tension');

out = model;

MatCom.sh

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


#!/bin/bash
#SBATCH --time=0-00:30:00
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --job-name=cube_tension
#SBATCH --output=cube_tension.out
#SBATCH --mem-per-cpu=2G
#SBATCH --partition=short

#SBATCH --error=slurm-%j.stderr
#SBATCH --mail-type=ALL
#SBATCH --mail-user=youremail@rug.nl

module purge

module load MATLAB/2019b

module load COMSOL/5.5

# Start the COMSOL server, usually on port 2036, if it is available, otherwise it will generate an error
comsol mphserver -np ${SLURM_CPUS_PER_TASK} -port 2036 -silent -tmpdir $TMPDIR &

# Run a MATLAB script called "test.m" which uses some COMSOL functions
matlab -nodisplay -r cube_tension