Deploying MATLAB Executables with Release R2006a
Deploying MATLAB Executables under MATLAB
R2006a
Assuming you have figured out how to compile MATLAB, the following shows
you how to deploy MATLAB executables for release R2006a on LSC grid computers, and
the Tier 2 centers at PSU and UWM.
If you haven't setup up MATLAB compiling, go to
this web page on MATLAB compilation.
The technique for deploying MATLAB executables on stand-alone grid computers with
independent disk systems (such as Penn State's Pleiades and UWM's Nemo) can differ from that
on grid computers (such as those on the main LSC grids CIT, LHO and LLO) that share disks with
the development nodes where a MATLAB installation resides
Deploying executables on stand-alone grids (PSU,UWM) or
Deploying executables on main LSC grids
Deploying executables on stand-alone grids (PSU,UWM)
To run MATLAB executables on grid computers without MATLAB installed (or without access to the MATLAB installation files),
we need to install the MATLAB Component Runtime (MCR) files on that system.
Such systems include Pleiades at PSU and Nemo at UWM.
1) Check if MCR is already set up
Often, the Grid computer sys-admin has already installed this. If that is the case, obtain the location of the MCR
installation from them and jump to step 4). On the Pleiades cluster, the MCR is installed at /usr1/psurg/MCR/v74.
On the Nemo cluster, the MCR is installed at /matlab/v74.
2) Get MCR Installer file (MCRInstaller.zip)
This can be created using the buildmcr
command within MATLAB. Because the user may not be allowed to
write into the default MATLAB directory, we have it create that file
locally. Use the following at the MATLAB prompt on the system (such as PSU's Gravity cluster) where you are compiling your executables
%
buildmcr('.');
This will take some time. A copy of the 32-bit glnx86 installer is stored on the Gravity cluster at
~s4/no-backup/Matlab_MCR/v74/glnx86.
A copy of the 64-bit glnxa64 is stored on the Gravity cluster at
~s4/no-backup/Matlab_MCR/v74/glnxa64.
3) Transfer and install on Grid computer
In the following, it is assumed that you can access the Grid computer using grid-enabled
ssh and sftp. For this example, I will use the Pleiades cluster. First, access the
grid computer head node with 'sftp' and create a directory for the MCR files
[thorne@auriga ~]$ cd ~s4/no-backup/Matlab_MCR/v74
[thorne@auriga v74]$ sftp pleiades.aset.psu.edu
Connecting to pleiades.aset.psu.edu...
sftp> mkdir MCR
2) Copy the MCRInstaller.zip for v7.4 to the grid computer and quit sftp
sftp> put MCRInstaller.zip
Uploading MCRInstaller.zip to /usr1/home/ux001007/MCRInstaller.zip
MCRInstaller.zip 100% 104MB 8.6MB/s 00:12
sftp> quit
[thorne@auriga v74]$
3) Access the grid computer head node with 'ssh' and unzip the MCR Installer on the grid computer
[thorne@auriga v74]$ ssh pleiades.aset.psu.edu
Last login: Thu Mar 2 11:54:30 2006 from auriga.gravity.psu.edu
[ux001007@pleiades ~]$ unzip MCRInstaller.zip -d MCR
[ux001007@pleiades ~]$ logout
[thorne@auriga v74]$
** this creates a v74 sub-directory under the MCR directory
4) Put script with MCR in LD_LIBRARY_PATH on Grid computer
To use MCR on the grid computer, the MCR needs to be added to the LD_LIBRARY_PATH.
This is similar to the setup above, but for MCR directories. Here it is for glnx86 in csh:
# -- the MCR_ROOT location is system specific
setenv MCR_ROOT /usr1/psurg/MCR/v74
setenv ARCH glnx86
if ( $?LD_LIBRARY_PATH ) then
setenv LD_LIBRARY_PATH ${MCR_ROOT}/sys/opengl/lib/glnx86:${LD_LIBRARY_PATH}
else
setenv LD_LIBRARY_PATH ${MCR_ROOT}/sys/opengl/lib/glnx86
endif
setenv LD_LIBRARY_PATH ${MCR_ROOT}/sys/java/jre/glnx86/jre1.5.0/lib/i386:${LD_LIBRARY_PATH}
setenv LD_LIBRARY_PATH ${MCR_ROOT}/sys/java/jre/glnx86/jre1.5.0/lib/i386/client:${LD_LIBRARY_PATH}
setenv LD_LIBRARY_PATH ${MCR_ROOT}/sys/java/jre/glnx86/jre1.5.0/lib/i386/native_threads:${LD_LIBRARY_PATH}
setenv LD_LIBRARY_PATH ${MCR_ROOT}/bin/glnx86:${LD_LIBRARY_PATH}
setenv LD_LIBRARY_PATH ${MCR_ROOT}/sys/os/glnx86:${LD_LIBRARY_PATH}
setenv LD_LIBRARY_PATH ${MCR_ROOT}/runtime/glnx86:${LD_LIBRARY_PATH}
setenv XAPPLRESDIR ${MCR_ROOT}/X11/app-defaults
This setup has some changes from the MathWorks instructions:
- The ARCH environment variable is defined. This is needed to fix a problem in calling Ghostscript
when creating figures in compiled code.
- The bin/glnx86 library is added to LD_LIBRARY_PATH. This catches some dependencies
that can be missed by the compiler
- The sys/opengl/lib/glnx86 library is also added. This again catches some dependencies
missed in the past by the compiler
Here it is for glnxa64 in csh:
# -- the MCR_ROOT location is system specific
setenv MCR_ROOT /matlab/v74
setenv ARCH glnxa64
if ( $?LD_LIBRARY_PATH ) then
setenv LD_LIBRARY_PATH ${MCR_ROOT}/sys/opengl/lib/glnxa64:${LD_LIBRARY_PATH}
else
setenv LD_LIBRARY_PATH ${MCR_ROOT}/sys/opengl/lib/glnxa64
endif
setenv LD_LIBRARY_PATH ${MCR_ROOT}/sys/java/jre/glnxa64/jre1.5.0/lib/amd64:${LD_LIBRARY_PATH}
setenv LD_LIBRARY_PATH ${MCR_ROOT}/sys/java/jre/glnxa64/jre1.5.0/lib/amd64/server:${LD_LIBRARY_PATH}
setenv LD_LIBRARY_PATH ${MCR_ROOT}/sys/java/jre/glnxa64/jre1.5.0/lib/amd64/native_threads:${LD_LIBRARY_PATH}
setenv LD_LIBRARY_PATH ${MCR_ROOT}/bin/glnxa64:${LD_LIBRARY_PATH}
setenv LD_LIBRARY_PATH ${MCR_ROOT}/sys/os/glnxa64:${LD_LIBRARY_PATH}
setenv LD_LIBRARY_PATH ${MCR_ROOT}/runtime/glnxa64:${LD_LIBRARY_PATH}
setenv XAPPLRESDIR ${MCR_ROOT}/X11/app-defaults
This setup has even more changes from the MathWorks instructions:
- Change the JRE from 1.4.2 to 1.5.0 (the MATLAB distribution came with 1.5.0)
- Change the JRE sub-directory lib/amd64/client to lib/amd64/server. The expected
'client' sub-directory was not in the distribution. This holds the Java Virtual
machine (JVM). In testing, I found the 'server' sub-directory also had a JVM.
I have prepared MDC setup scripts for the following:
You should be able to modify these as needed.
Transfer this MCR Setup file to the Grid computer using sftp. Here is an example
for Pleiades.
[thorne@auriga ~]$ sftp pleiades.aset.psu.edu
Connecting to pleiades.aset.psu.edu...
sftp> put MCRSetup_R2006a.csh
5) Modify LDG setup to avoid PATH > 1024 problem
There are some built-in MATLAB functions that at present need the PATH environment variable
to be fewer that 1,024 characters. The symptom is bizarre segmentation fault errors
in relatively innocuous code. The culprit is the LDG initialization, which inflates
the path to > 1024 characters with un-needed, admin-only directories. There are
solutions available. On the LSC grid computers, Nick Fotopoulos has the following
work-around to add to your .cshrc file.
# turn off LDG setup
touch ~/.noldg
# LDG 4.0
# setup python for LSCdataFind and LSCsegFind
setenv PYTHONPATH /ldcg/stow_pkgs/ldg-4.0/ldg/vdt/globus/lib/python:${PYTHONPATH}
# setup grid-enabled SSH
setenv PATH /ldcg/stow_pkgs/ldg-4.0/ldg/vdt/globus/bin:${PATH}
For the lazy I have prepared a .bashrc fragment. This
is also available as a .cshrc fragment. You may
find that you need other LDG pieces. A longer-term solution is to tear the
'sbin' paths out of the LDG setup, as these are only needed by LDG admins.
On the Pleiades grid computer, LDG is automatically setup. Shantanu Desai has developed
a different work-around that also forces the LDG version. There is a
csh code fragment available, but this fix is already
included in the PSU MCR setup file found above
6) Transfer MATLAB executable to Grid computer
You must move both the compiled executable and the .ctf file to
the Grid computer. If doing it statically and assuming 'matlabprog' is the
executable, then use
sftp> mkdir Exectuables
sftp> cd Executables
sftp> put matlabprog
sftp> put matlabprog.ctf
sftp> chmod 755 matlabprog
sftp> quit
An alternative is to transfer these within the Condor submission, but remember
that you still need to do the 'chmod 755' on the transferred MATLAB executable.
7) Create script to run MATLAB executable on grid
I have found it easier to use a shell script as the 'executable' under condor. Since
you can place the MATLAB compiled executable in directories visible from the condor
grid nodes, your script can concentrate on setup concerns. Assuming
a) your home directory is /usr1/home/griduser on ligo-grid.aset.psu.edu
b) your MATLAB exectuable is named 'matlabprog' and is in sub-directory 'Executables'
c) you have put the MCRSetup.csh script in your home directory on the grid computer
then the following is needed in a script to run your executable:
(the #'s denote optional comment lines)
# define the home directory (this is done automatically on Pleiades)
setenv HOME /usr1/home/griduser
# add MATLAB MCR to your LD_LIBRARY_PATH
source ${HOME}/MCRSetup_R2006a.csh
# add directory with executable to LD_LIBRARY_PATH
setenv LD_LIBRARY_PATH ${HOME}/Executables:${LD_LIBRARY_PATH}
# define variable to point at your executable
setenv MATBIN ${HOME}/Executables/matlabprog
# run your executable
${MATBIN} ..command-line arguments..
Deploying executables on LSG grid computers
On the main LSC grid computers (CIT, LHO, LLO), the processing nodes can see (through NFS)
the same directories as the head and development nodes. This means that MATLAB executables
do not have to be tranferred to the "grid computer". Also, since the installed MATLAB
libraries are visible, there is no need to build and deploy a MATLAB Common Runtime (MCR).
1) Limit PATH to < 1024 characters
There is a nasty bug in compiled MATLAB that causes segmentation faults,etc. when calling
some built-in functions. This occurs if your PATH has more that 1024 characters. The main
culprit in this is the LDG initialization. Nick Fotopoulos has developed a workaround
by only doing the LDG initialization actually needed. In your .bashrc, add the following
# turn off LDG setup
touch ~/.noldg
# LDG 4.0
# setup python for LSCdataFind and LSCsegFind
setenv PYTHONPATH /ldcg/stow_pkgs/ldg-4.0/ldg/vdt/globus/lib/python:${PYTHONPATH}
# setup grid-enabled SSH
setenv PATH /ldcg/stow_pkgs/ldg-4.0/ldg/vdt/globus/bin:${PATH}
For the lazy I have prepared a .bashrc fragment. This
is also available as a .cshrc fragment. You may
find that you need other LDG pieces. A longer-term solution is to tear the
'sbin' paths out of the LDG setup, as these are only needed by LDG admins.
2) Make sure HOME and PATH will be defined for grid jobs
When MATLAB or the MCR expands a *.ctf file, it wants to save a preferences file to
the location ${HOME}/.matlab/mcr_v74. If HOME is not a defined environment variable,
it will fail. Also, it requires that the LD_LIBRARY_PATH and PATH environment variables
be defined.
One way to ensure this with Condor grid jobs is to include 'getenv = True' in the condor
submission. Add the following in your Condor submission script
getenv = True
3) Fix your startup.m file to running executables
When running a stand-alone build on a system where the grid nodes can see the
development disks (such as CIT,LLO,LHO), you need to make sure it sees
the local encrypted m-files instead of ones from your CVS sandbox.
We wrap any addpath
commands (which let us see the CVS sandbox files) with a conditional to
prevent their use when running an
executable. For Release 2006a, the
version('-release') function returns
a text string 2006a that
can not be converted to a numeric value. This complicates things if you wish to preserve backward compatibility
with Release 13. A complete syntax of the startup.m file should now be:
verNum = str2num(version('-release'));
if(~isempty(verNum))
verOld = verNum < 14;
else
verOld = false;
end
if( (verOld == true) || (~isdeployed))
...
addpath commands
...
end
clear verNum;
clear verOld;
The startup.m file should be in your ~/matlab
directory.
**NOTE** You may have already done this if following the compilation info page
4A) Create 32-bit MATLAB setup script
If using 32-bit executables, a script is needed to to add the 32-bit 'glnx86' MATLAB libraries to
LD_LIBRARY_PATH when running the MATLAB executables. This script can be the same one used when testing your compiled
MATLAB.
If you don't already have such a script, create a script with the following:
setenv MATLAB_ROOT /ldcg/matlab_r2006a
setenv ARCH glnx86
#
if ( $?LD_LIBRARY_PATH ) then
setenv LD_LIBRARY_PATH ${MATLAB_ROOT}/sys/opengl/lib/glnx86:${LD_LIBRARY_PATH}
else
setenv LD_LIBRARY_PATH ${MATLAB_ROOT}/sys/opengl/lib/glnx86
endif
setenv LD_LIBRARY_PATH ${MATLAB_ROOT}/sys/java/jre/glnx86/jre1.5.0/lib/i386:${LD_LIBRARY_PATH}
setenv LD_LIBRARY_PATH ${MATLAB_ROOT}/sys/java/jre/glnx86/jre1.5.0/lib/i386/client:${LD_LIBRARY_PATH}
setenv LD_LIBRARY_PATH ${MATLAB_ROOT}/sys/java/jre/glnx86/jre1.5.0/lib/i386/native_threads:${LD_LIBRARY_PATH}
setenv LD_LIBRARY_PATH ${MATLAB_ROOT}/bin/glnx86:${LD_LIBRARY_PATH}
setenv LD_LIBRARY_PATH ${MATLAB_ROOT}/sys/os/glnx86:${LD_LIBRARY_PATH}
setenv XAPPLRESDIR ${MATLAB_ROOT}/X11/app-defaults
#
unsetenv MATLAB
Note that this is in 'reverse' order from the Compiler list to ensure correct order
in LD_LIBRARY_PATH. Also, I have added a library (/sys/opengl/lib/glnx86) which previous experience
with Compiler 4.1 showed was often necessary.
I have provided an example MatlabSetup_R2006a_glnx86.csh script which handles a non-existent LD_LIBRARY_PATH and checks if MATLAB was already added to it. This is the same as the one used in testing executables
on the development node.
4B) Create 64-bit MATLAB setup script
If using 64-bit executables at CIT, a script is needed to to add the 64-bit 'glnxa64' MATLAB
libraries to LD_LIBRARY_PATH
when running the MATLAB executables. This script can be the same one used when testing your
compiled MATLAB. If you don't already have such a script, create a script with the following:
setenv MATLAB_ROOT /ldcg/matlab_r2006a
setenv ARCH glnxa64
#
if ( $?LD_LIBRARY_PATH ) then
setenv LD_LIBRARY_PATH ${MATLAB_ROOT}/sys/opengl/lib/glnxa64:${LD_LIBRARY_PATH}
else
setenv LD_LIBRARY_PATH ${MATLAB_ROOT}/sys/opengl/lib/glnxa64
endif
setenv LD_LIBRARY_PATH ${MATLAB_ROOT}/sys/java/jre/glnxa64/jre1.5.0/lib/amd64:${LD_LIBRARY_PATH}
setenv LD_LIBRARY_PATH ${MATLAB_ROOT}/sys/java/jre/glnxa64/jre1.5.0/lib/amd64/server:${LD_LIBRARY_PATH}
setenv LD_LIBRARY_PATH ${MATLAB_ROOT}/sys/java/jre/glnxa64/jre1.5.0/lib/amd64/native_threads:${LD_LIBRARY_PATH}
setenv LD_LIBRARY_PATH ${MATLAB_ROOT}/sys/os/glnxa64:${LD_LIBRARY_PATH}
setenv LD_LIBRARY_PATH ${MATLAB_ROOT}/bin/glnxa64:${LD_LIBRARY_PATH}
setenv XAPPLRESDIR ${MATLAB_ROOT}/X11/app-defaults
#
unsetenv MATLAB
Note that this is in 'reverse' order from the Compiler list to ensure correct order
in LD_LIBRARY_PATH. This setup does not match that in the MATLAB documentation.
I have had to
- Add a library (/sys/opengl/lib/glnxa64) which previous experience with Compiler 4.3
showed was often necessary.
- Change the JRE from 1.4.2 to 1.5.0 (the MATLAB distribution came with 1.5.0)
- Change the JRE sub-directory lib/amd64/client to lib/amd64/server. The expected
'client' sub-directory was not in the distribution. This holds the Java Virtual
machine (JVM). In testing,
I found the 'server' sub-directory also had a JVM, and code that was compiled
without the JVM disabled ran successfully.
I have provided an example
MatlabSetup_R2006a_glnxa64.csh which handles
a non-existent LD_LIBRARY_PATH and checks if MATLAB was already added to it.
5A) Create script to run 32-bit MATLAB executable on grid
I have found it easier to use a shell script as the 'executable' under condor. Since
you can place the MATLAB compiled executable in directories visible from the condor
grid nodes, your script can concentrate on setup concerns. Assuming
a) your home directory is /archive/home/griduser on ldas-grid.ligo-*.caltech.edu
b) your MATLAB exectuable is named 'matlabprog' and is in directory /archive/home/griduser/Executables
c) you have put the MatlabSetup_R2006a_glnx86.csh script in your home directory
then the following is needed in a script to run your 32-bit executable:
(the #'s denote optional comment lines)
# define the home directory (this is not set automatically)
# MATLAB needs HOME defined.
setenv HOME /archive/home/griduser
# add 32-bit MATLAB MCR to your LD_LIBRARY_PATH
source ${HOME}/MatlabSetup_R2006a_glnx86.csh
# add directory with executable to LD_LIBRARY_PATH
setenv LD_LIBRARY_PATH ${HOME}/Executables:${LD_LIBRARY_PATH}
# define variable to point at your executable
setenv MATBIN ${HOME}/Executables/matlabprog
# run your executable
${MATBIN} ..command-line arguments..
5B) Create script to run 64-bit MATLAB executable on grid
I have found it easier to use a shell script as the 'executable' under condor. Since
you can place the MATLAB compiled executable in directories visible from the condor
grid nodes, your script can concentrate on setup concerns. Assuming
a) your home directory is /archive/home/griduser on ldas-grid.ligo.caltech.edu
b) your MATLAB exectuable is named 'matlabprog' and is in directory /archive/home/griduser/Executables
c) you have put the MatlabSetup_R2006a_glnxa64.csh script in your home directory
then the following is needed in a script to run your 64-bit executable:
(the #'s denote optional comment lines)
# define the home directory (this is not set automatically)
# MATLAB needs HOME defined.
setenv HOME /archive/home/griduser
# add 64-bit MATLAB MCR to your LD_LIBRARY_PATH
source ${HOME}/MatlabSetup_R2006a_glnxa64.csh
# add directory with executable to LD_LIBRARY_PATH
setenv LD_LIBRARY_PATH ${HOME}/Executables:${LD_LIBRARY_PATH}
# define variable to point at your executable
setenv MATBIN ${HOME}/Executables/matlabprog
# run your executable
${MATBIN} ..command-line arguments..
Keith Thorne t h o r n e @ g r a v i t y . p s u . e d u
Last modified: Wed, Aug 23, 2006