Deploying MATLAB Executables with Release 14 SP3

Deploying MATLAB Executables under MATLAB R14 SP3

Assuming you have figured out how to compile MATLAB, the following shows you how to deploy MATLAB executables on LSC grid computers.
If you haven't setup up MATLAB compiling, go to this web page on MATLAB compilation.
The technique for deploying MATLAB executables on stand-alone grid computers with independent disk systems (such as Penn State's Pleiades) can differ from that on grid computers (such as those on the main LSC grids CIT, LHO and LLO) that share disks with the development nodes where a MATLAB installation resides
Deploying executables on stand-alone grids (Pleiades) or
Deploying executables on main LSC grids

Deploying executables on stand-alone grids (Pleiades)

To run MATLAB executables on grid computers without MATLAB installed (or without access to the MATLAB installation files), we need to install the MATLAB Component Runtime (MCR) files on that system. Such systems include Pleiades and other LSC Tier 2 computers.

1) Check if MCR is already set up

Often, the Grid computer sys-admin has already installed this. If that is the case, obtain the location of the MCR installation from them and jump to step 4)

2) Get MCR Installer file (MCRInstaller.zip)

This can be created using the buildmcr command within MATLAB. Because the user may not be allowed to write into the default MATLAB directory, we have it create that file locally. Use the following at the MATLAB prompt on the system (such as PSU's Gravity cluster) where you are compiling your executables
% buildmcr('.');
This will take some time. A copy of the installer is stored on the Gravity cluster at ~s4/no-backup/Matlab_MCR/v73.

3) Transfer and install on Grid computer

In the following, it is assumed that you can access the Grid computer using grid-enabled ssh and sftp. For this example, I will use the Pleiades cluster. First, access the grid computer head node with 'sftp' and create a directory for the MCR files

[thorne@auriga ~]$ cd ~s4/no-backup/Matlab_MCR/v73
[thorne@auriga v73]$ sftp pleiades.aset.psu.edu
Connecting to pleiades.aset.psu.edu...
sftp> mkdir MCR

2) Copy the MCRInstaller.zip for v7.3 to the grid computer and quit sftp

sftp> put MCRInstaller.zip
Uploading MCRInstaller.zip to /usr1/home/ux001007/MCRInstaller.zip
MCRInstaller.zip                              100%  104MB   8.6MB/s   00:12
sftp> quit
[thorne@auriga v73]$

3) Access the grid computer head node with 'ssh' and unzip the MCR Installer on the grid computer

[thorne@auriga v73]$ ssh pleiades.aset.psu.edu
Last login: Thu Mar  2 11:54:30 2006 from auriga.gravity.psu.edu
[ux001007@pleiades ~]$ unzip MCRInstaller.zip -d MCR
[ux001007@pleiades ~]$ logout
[thorne@auriga v73]$

** this creates a v73 sub-directory under the MCR directory

4) Put script with MCR in LD_LIBRARY_PATH on Grid computer

To use MCR on the grid computer, the MCR needs to be added to the LD_LIBRARY_PATH.
This is similar to the setup above, but for MCR directories. Here it is for csh:

#  ** Force load of LDG 3.5
alias LSCdataFind  '/usr1/global/ldg-3.5/glue/bin/LSCdataFind'
setenv GLUE_LOCATION /usr1/global/ldg-3.5/glue
source /usr1/global/ldg-3.5/glue/etc/glue-user-env.csh
source /usr1/global/ldg-3.5/setup.csh
#  ** Override PATH definition bloat from LDG to keep character count < 1024
unsetenv $PATH
setenv PATH /usr1/global/ldg-3.5/jdk1.4/bin:/usr1/global/ldg-3.5/pyglobus-url-copy/bin:
setenv PATH ${PATH}/usr1/global/ldg-3.5/netlogger/bin:/usr1/global/ldg-3.5/ftsh/bin:/usr1/global/ldg-3.5/condor/bin:
setenv PATH ${PATH}/usr1/global/ldg-3.5/logrotate/bin:/usr1/global/ldg-3.5/globus/bin:/usr1/global/ldg-3.5/vdt/bin:
setenv PATH ${PATH}/usr1/global/ldg-3.5/ldg-client/bin:/usr1/global/ldg-3.5/glue/bin:/usr/global/hdf5/5-1.6.4-linux-gcc/bin:
setenv PATH ${PATH}/usr1/global/lscdatagrid/python/bin:/usr/kerberos/bin:/bin:/usr/bin:/usr/X11R6/bin:/usr/pbs/bin:
setenv PATH ${PATH}/usr/global/intel//bin:/usr/global/mysql/bin:/usr/global/lib:/usr/global/lib/psurg:/usr/global/R-1.9.0/bin:
setenv PATH ${PATH}/usr/global/mpich.intel.2/bin:/usr/global/lam/bin:/usr/global/fortran/bin:/usr/global/ligotools/bin:
setenv PATH ${PATH}/usr1/global/ldg-3.5/vds/bin:/usr1/global/ldg-3.5/vds/contrib/gstar/bin
#
setenv MCR_ROOT ${HOME}/MCR/v73
setenv ARCH glnx86
if ( $?LD_LIBRARY_PATH ) then
    setenv LD_LIBRARY_PATH ${MCR_ROOT}/sys/opengl/lib/glnx86:${LD_LIBRARY_PATH}
else
    setenv LD_LIBRARY_PATH ${MCR_ROOT}/sys/opengl/lib/glnx86
endif
setenv LD_LIBRARY_PATH ${MCR_ROOT}/sys/java/jre/glnx86/jre1.5.0/lib/i386:${LD_LIBRARY_PATH}
setenv LD_LIBRARY_PATH ${MCR_ROOT}/sys/java/jre/glnx86/jre1.5.0/lib/i386/client:${LD_LIBRARY_PATH}
setenv LD_LIBRARY_PATH ${MCR_ROOT}/sys/java/jre/glnx86/jre1.5.0/lib/i386/native_threads:${LD_LIBRARY_PATH}
setenv LD_LIBRARY_PATH ${MCR_ROOT}/sys/os/glnx86:${LD_LIBRARY_PATH}
setenv LD_LIBRARY_PATH ${MCR_ROOT}/runtime/glnx86:${LD_LIBRARY_PATH}
setenv XAPPLRESDIR ${MCR_ROOT}/X11/app-defaults

Note the extra LDG material at the top. One can have problems ensuring the desired LDG version is loaded on a node. This forces it. The PATH part is because of the nasty compiled MATLAB bug causing segmentation faults, etc. if the PATH variable exceeds 1024 characters. The usual culprit is the LDG setup. The workaround above (courtesy of Shantanu Desai) is one solution. This setup is actually slightly different than that used when testing with a MATLAB installation (here the 'runtime' directory is used). Note the addition of ARCH. This is needed to fix a problem in calling Ghostscript when creating figures in compiled code. There is a sample csh file (MCRSetup.csh) available to use as well. Transfer this file to the Grid computer using sftp

[thorne@auriga ~]$ sftp pleiades.aset.psu.edu
Connecting to pleiades.aset.psu.edu...
sftp> put MCRSetup.csh

5) Transfer MATLAB executable to Grid computer

You must move both the compiled executable and the .ctf file to the Grid computer. If doing it statically and assuming 'matlabproc' is the executable, then use

sftp> mkdir Exectuables
sftp> cd Executables
sftp> put matlabproc
sftp> put matlabproc.crf
sftp> chmod 755 matlabproc
sftp> quit

An alternative is to transfer these within the Condor submission, but remember that you still need to do the 'chmod 755' on the transferred MATLAB executable.

5) Create script to run MATLAB executable on grid

I have found it easier to use a shell script as the 'executable' under condor. Since you can place the MATLAB compiled executable in directories visible from the condor grid nodes, your script can concentrate on setup concerns. Assuming
a) your home directory is /usr1/home/griduser on ligo-grid.aset.psu.edu
b) your MATLAB exectuable is named 'matlabprog' and is in sub-directory 'Executables'
c) you have put the MCRSetup.csh script in your home directory on the grid computer
then the following is needed in a script to run your executable: (the #'s denote optional comment lines)

# define the home directory (this is done automatically on Pleiades)
setenv HOME /usr1/home/griduser
# add MATLAB MCR to your LD_LIBRARY_PATH
source ${HOME}/MCRSetup_v73.csh
# add directory with executable to LD_LIBRARY_PATH
setenv LD_LIBRARY_PATH ${HOME}/Executables:${LD_LIBRARY_PATH}
# define variable to point at your executable
setenv MATBIN ${HOME}/Executables/matlabprog
#  run your executable
${MATBIN} ..command-line arguments..

Deploying executables on LSG grid computers

On the main LSC grid computers (CIT, LHO, LLO), the processing nodes can see (through NFS) the same directories as the head and development nodes. This means that MATLAB executables do not have to be tranferred to the "grid computer". Also, since the installed MATLAB libraries are visible, there is no need to build and deploy a MATLAB Common Runtime (MCR).

1) Limit PATH to < 1024 characters

There is a nasty bug in compiled MATLAB that causes segmentation faults,etc. when calling some built-in functions. This occurs if your PATH has more that 1024 characters. The main culprit in this is the LDG initialization. Nick Fotopoulos has developed a workaround by only doing the LDG initialization actually needed. In your .bashrc, add the following

# turn off LDG setup
touch ~/.noldg
# setup python for LSCdataFind and LSCsegFind
export PYTHONPATH="${PYTHONPATH}:/ldcg/stow-pkgs/ldg-3.5/ldg/pyGlobus/lib/python"
# setup grid-enabled SSH
export PATH="/ldcg/stow-pkgs/ldg-3.5/ldg/globus/bin:${PATH}"

For the lazy I have prepared a .bashrc fragment. This is also available as a .cshrc fragment. You may find that you need other LDG pieces. A longer-term solution is to tear the 'sbin' paths out of the LDG setup, as these are only needed by LDG admins.

2) Make sure HOME and PATH will be defined for grid jobs

When MATLAB or the MCR expands a *.ctf file, it wants to save a preferences file to the location ${HOME}/.matlab/mcr_v73. If HOME is not a defined environment variable, it will fail. Also, it requires that the LD_LIBRARY_PATH and PATH environment variables be defined.

One way to ensure this with Condor grid jobs is to include 'getenv = True' in the condor submission. Add the following in your Condor submission script

getenv = True

3) Fix your startup.m file to running executables

When running a stand-alone build on a system where the grid nodes can see the development disks (such as CIT,LLO,LHO), you need to make sure it sees the local encrypted m-files instead of ones from your CVS sandbox. We wrap any addpath commands (which let us see the CVS sandbox files) with a conditional to prevent their use when running an executable. The syntax of the startup.m file should be:

if((str2num(version('-release')) < 14) || ~isdeployed)
   addpath commands
   ...
end

The startup.m file should be in your ~/matlab directory.

**NOTE** You may have already done this if following the compilation info page

4A) Create 32-bit MATLAB setup script

If using 32-bit executables, a script is needed to to add the 32-bit 'glnx86' MATLAB libraries to LD_LIBRARY_PATH when running the MATLAB executables. This script can be the same one used when testing your compiled MATLAB. If you don't already have such a script, create a script with the following:

setenv MATLAB_ROOT /ldcg/matlab_r14_sp3
setenv ARCH glnx86
#
if ( $?LD_LIBRARY_PATH ) then
    setenv LD_LIBRARY_PATH ${MATLAB_ROOT}/sys/opengl/lib/glnx86:${LD_LIBRARY_PATH}
else
    setenv LD_LIBRARY_PATH ${MATLAB_ROOT}/sys/opengl/lib/glnx86
endif
setenv LD_LIBRARY_PATH ${MATLAB_ROOT}/sys/java/jre/glnx86/jre1.5.0/lib/i386:${LD_LIBRARY_PATH}
setenv LD_LIBRARY_PATH ${MATLAB_ROOT}/sys/java/jre/glnx86/jre1.5.0/lib/i386/client:${LD_LIBRARY_PATH}
setenv LD_LIBRARY_PATH ${MATLAB_ROOT}/sys/java/jre/glnx86/jre1.5.0/lib/i386/native_threads:${LD_LIBRARY_PATH}
setenv LD_LIBRARY_PATH ${MATLAB_ROOT}/bin/glnx86:${LD_LIBRARY_PATH}
setenv LD_LIBRARY_PATH ${MATLAB_ROOT}/sys/os/glnx86:${LD_LIBRARY_PATH}
setenv XAPPLRESDIR ${MATLAB_ROOT}/X11/app-defaults
#
unsetenv MATLAB

Note that this is in 'reverse' order from the Compiler list to ensure correct order in LD_LIBRARY_PATH. Also, I have added a library (/sys/opengl/lib/glnx86) which previous experience with Compiler 4.1 showed was often necessary. I have provided an example MatlabSetup_32bit.csh script which handles a non-existent LD_LIBRARY_PATH and checks if MATLAB was already added to it. This is the same as the one used in testing executables on the development node.

4B) Create 64-bit MATLAB setup script

If using 64-bit executables at CIT, a script is needed to to add the 64-bit 'glnxa64' MATLAB libraries to LD_LIBRARY_PATH when running the MATLAB executables. This script can be the same one used when testing your compiled MATLAB. If you don't already have such a script, create a script with the following:

setenv MATLAB_ROOT /ldcg/matlab_r14_sp3
setenv ARCH glnxa64
#
if ( $?LD_LIBRARY_PATH ) then
    setenv LD_LIBRARY_PATH ${MATLAB_ROOT}/sys/opengl/lib/glnxa64:${LD_LIBRARY_PATH}
else
    setenv LD_LIBRARY_PATH ${MATLAB_ROOT}/sys/opengl/lib/glnxa64
endif
setenv LD_LIBRARY_PATH ${MATLAB_ROOT}/sys/java/jre/glnxa64/jre1.4.2/lib/amd64:${LD_LIBRARY_PATH}
setenv LD_LIBRARY_PATH ${MATLAB_ROOT}/sys/java/jre/glnxa64/jre1.4.2/lib/amd64/client:${LD_LIBRARY_PATH}
setenv LD_LIBRARY_PATH ${MATLAB_ROOT}/sys/java/jre/glnxa64/jre1.4.2/lib/amd64/native_threads:${LD_LIBRARY_PATH}
setenv LD_LIBRARY_PATH ${MATLAB_ROOT}/sys/os/glnxa64:${LD_LIBRARY_PATH}
setenv LD_LIBRARY_PATH ${MATLAB_ROOT}/bin/glnxa64:${LD_LIBRARY_PATH}
setenv XAPPLRESDIR ${MATLAB_ROOT}/X11/app-defaults
#
unsetenv MATLAB

Note that this is in 'reverse' order from the Compiler list to ensure correct order in LD_LIBRARY_PATH. Also, I have added a library (/sys/opengl/lib/glnxa64) which previous experience with Compiler 4.1 showed was often necessary. I have provided an example MatlabSetup_64bit.csh which handles a non-existent LD_LIBRARY_PATH and checks if MATLAB was already added to it.

5A) Create script to run 32-bit MATLAB executable on grid

I have found it easier to use a shell script as the 'executable' under condor. Since you can place the MATLAB compiled executable in directories visible from the condor grid nodes, your script can concentrate on setup concerns. Assuming
a) your home directory is /archive/home/griduser on ldas-grid.ligo-*.caltech.edu
b) your MATLAB exectuable is named 'matlabprog' and is in directory /archive/home/griduser/Executables
c) you have put the MatlabSetup_32bit.csh script in your home directory
then the following is needed in a script to run your 32-bit executable: (the #'s denote optional comment lines)

# define the home directory (this is not set automatically)
#  MATLAB needs HOME defined.
setenv HOME /archive/home/griduser
# add 32-bit MATLAB MCR to your LD_LIBRARY_PATH
source ${HOME}/MatlabSetup_32bit.csh
# add directory with executable to LD_LIBRARY_PATH
setenv LD_LIBRARY_PATH ${HOME}/Executables:${LD_LIBRARY_PATH}
# define variable to point at your executable
setenv MATBIN ${HOME}/Executables/matlabprog
#  run your executable
${MATBIN} ..command-line arguments..

5B) Create script to run 64-bit MATLAB executable on grid

I have found it easier to use a shell script as the 'executable' under condor. Since you can place the MATLAB compiled executable in directories visible from the condor grid nodes, your script can concentrate on setup concerns. Assuming
a) your home directory is /archive/home/griduser on ldas-grid.ligo.caltech.edu
b) your MATLAB exectuable is named 'matlabprog' and is in directory /archive/home/griduser/Executables
c) you have put the MatlabSetup_64bit.csh script in your home directory
then the following is needed in a script to run your 64-bit executable: (the #'s denote optional comment lines)

# define the home directory (this is not set automatically)
#  MATLAB needs HOME defined.
setenv HOME /archive/home/griduser
# add 64-bit MATLAB MCR to your LD_LIBRARY_PATH
source ${HOME}/MatlabSetup_64bit.csh
# add directory with executable to LD_LIBRARY_PATH
setenv LD_LIBRARY_PATH ${HOME}/Executables:${LD_LIBRARY_PATH}
# define variable to point at your executable
setenv MATBIN ${HOME}/Executables/matlabprog
#  run your executable
${MATBIN} ..command-line arguments..

Keith Thorne

Last modified: Mon, Mar 6 2006