[ISyE Helpdesk] Recently Added Articles

Why did my job die?

Wed, 24 Apr 2013 12:58:49 GMT

There is no limit to the number of jobs you can submit. If there are no resources to run jobs, they will simply wait in the queue until resources are made available. There are a number of reasons jobs might be killed. The most common reason is that a job used more memory than was available on the machine. When this happens, the job will be killed in order to preserve the usability of the system. In addition, the job would most likely not have completed in a reasonable amount of time due to swapping to disk, having used up all available physical memory. Another possible reason is that the job ran on a private node and the owner logged in or submitted a job to that node. Private nodes, when not in use by their owners, are in the general cluster. When a node's owner logs into it, or submits jobs to it, community jobs on the node will be pre-empted in favor of the owner's jobs. The final reason for a job being killed is that a parallel job was submitted and your job was running on the boyles/wilkins and took longer than 5 calendar days to complete. These are the parallel machines and will kill serial jobs in favor of a parallel job, but they do have a grace period of 5 calendar days in order to allow the serial jobs to try to reach completion.

You can see your job history with the condor_history command. Like so:

condor_history <username>

You can then see the specifics about the job using the -long modifier with the job number. e.g.

condor_history -long <jobID>

That's the first place to start.

There is also the log in /tmp. That gives some very basic info about job statuses.

If you find more specific details, I may be able to help you interpret them.

Tracking time in condor

Mon, 25 Feb 2013 02:11:52 GMT

Condor tracks a number of different elapsed time counters. This page details the different time counters that condor maintains.

http://research.cs.wisc.edu/htcondor/manual/v7.6/26ManagingJob.html

Why is my job stuck in idle?

Mon, 25 Feb 2013 13:39:34 GMT

There are any number of reasons a job can remain idle. The first thing to do is check the job's output files, then use the command "condor_q -l" to get a long format job listing.

One common reason for idle jobs is running out of memory, which is usually due to memory leaks. The condor_q -l output contains the "ImageSize" attribute, expressed in KB. If this amount is larger than 12000000 (12GB) then you know you have a problem; and it could be a problem even at "only" 8GB.

For more detailed information, please see the appropriate section of the condorHT manual 2.6.5 Why is the job not running?

Running out of memory

Mon, 25 Feb 2013 21:19:06 GMT

Most of the time, if a program is compiled in C and it is using an unusually large amount of memory, then there is a memory leak. Unless you have some specific reason to expect a very large data set in memory, you should first attempt to find the memory leak.

In most cases, there are techniques you can utilize to optimize your code and reduce your memory footprint. You might also need to change your strategy for handling your computations. Perhaps you can slice your dataset and run more smaller jobs to achieve your results. This is going to give you much more redundancy and a better chance of completing your work, simply because you'll be less vulnerable to losing a lot of work when one job is somehow killed.

It is not unreasonable to need more than a couple of GB of memory these days, especially if you are using Matlab or another product which can add a fair amount of overhead. In those cases, you should run a test job to determine the size of your memory footprint. Run a job and then use the condor_history -long command to view the classed attributes of the job. Look for image_size. This is the maximum observed image size reported in KB. Use this as a guide for setting the image_size attribute in your command file. That attribute is set in KB as well. This attribute tells condor that your job may use up to the amount of memory specified. Condor will use this attribute to match appropriate resources so that you job will not run on a machine with too little memory.

How can I avoid pre-emption of my long-running jobs?

Fri, 30 Nov 2012 17:51:42 GMT

The best way to use Condor, when possible, is to break down large tasks into smaller ones and submit many small jobs rather than a few large ones. This makes preemption irrelevant because it's not a big deal if small jobs are restarted from the beginning.

For longer jobs, checkpointing is recommended.

If a job can't be easily broken down into smaller components, and it isn't checkpointed, then routine preemption can be prevented by running in the dedicated queue on hooke.

Why do my jobs keep restarting in condor?

Thu, 06 Dec 2012 13:56:09 GMT

Jobs in the normal Condor queues (those on Wren) are subject to preemption. When a job is preempted, it is vacated from the node and restarted elsewhere.

Jobs are usually stopped and restarted on Condor for one of four reasons:

Someone logs in locally to the execute machine. Many of our Condor compute nodes were bought by individual faculty members for their own research, and a direct shell login by one of their users will cause jobs to vacate.
You're running your job on the boyles and someone submitted an MPI job. While we have turned off priority pre-emption, parallel universe jobs will still pre-empt other jobs. The time to vacate is set at 5 calendar days. If your job does not complete within 5 calendar days, it will be vacated from the nodes in deference to the parallel universe job.
The job ran out of memory. Initially the process will be killed and the job will be restarted on another machine with more available memory. If memory usage keeps rising, eventually the job will just remain idle.
The node died while your job was running on it. This usually happens due to memory errors, which could be either yours or another user's.

Matlab in condor

Fri, 30 Nov 2012 17:48:05 GMT

Matlab must have threading disabled for use in the normal Condor queue. To do tihs, use the command line argument "-singleCompThread".

Under Condor we can achieve total higher compute throughput without threading, and will finish more jobs faster.

When launching Matlab in batch mode from the Unix shell, the program name is piped from STDIN rather than being an argument as in some other languages.

Thus for the Matlab program "program0.m", the command line for testing would be:

matlab -singleCompThread < program0.m

Then the Condor submit script would look like this:


universe = vanilla
executable  = /usr/local/bin/matlab
arguments   = -singleCompThread
input       = program0.m
output      = program0.out
error       = program0.err
queue

CPLEX in condor

Fri, 30 Nov 2012 17:45:57 GMT

Disabling threading in CPLEX

Since version 12.2, CPlex defaults to multithreading its executables.

This is a problem for several reasons and multithreading should be disabled in almost all cases. Not only does Condor handle single threaded jobs much better, but they will usually produce better performance.

We've modified the default environment in several places but it doesn't seem to work consistently. Thus users need to manually add a line to their C/C++ programs before compilation to turn off threading.

The C function call to disable threading should be put at the beginning of main():

CPXsetintparam(env, CPX_PARAM_THREADS, 1);

In an AMPL batch file the command to turn off multithreading is:

option cplex_options 'threads=1';

Timing processes

Fri, 30 Nov 2012 17:44:16 GMT

Determining process compute time is a common need in ISyE for such tasks as comparing algorithm efficiency. However, use of real time is likely to give erroneous results due to variations in compute environment, such as higher load and more context switching on some nodes than others. It is also possible that jobs can be suspended briefly without restarting from the beginning, which could greatly throw off compute times.

If you need to measure the compute time, you should use the USER time of the process to get the actual time it has been running on the processor.

In C, this value can be extracted with the getrusage() function.

The following example program runs three counting loops, but each subsequent step includes a longer sleep(). Nevertheless, the same elapsed time is reported for each step (within a millisecond). The function timeval_subtract() is taken from this GNU tutorial determining on elapsed CPU time.


#include 
#include 
#include  

int main() {
  int i, j, k;
  struct rusage usage;
  struct timeval current;
  struct timeval previous = {0, 0}; 

  for (i=0; i<3; i++)  {
     k=0;
     for (j=0; j<100000000; j++) {
        k++;
     }
     printf("Sleeping for %d seconds after counting\n", i);
     sleep(i);
     getrusage(RUSAGE_SELF, &usage);
     timeval_subtract (¤t, &usage.ru_utime, &previous);
     printf("Part %d took %d.%06d seconds of CPU time\n",i+1,  current.tv_sec,current.tv_usec);
     previous=usage.ru_utime;
  }
} 

int timeval_subtract (result, x, y)
struct timeval *result, *x, *y;
{ 

 /* Perform the carry for the later subtraction by updating y. */
  if (x->tv_usec < y->tv_usec) {
     int nsec = (y->tv_usec - x->tv_usec) / 1000000 + 1;
     y->tv_usec -= 1000000 * nsec;
     y->tv_sec += nsec;
  }
  if (x->tv_usec - y->tv_usec > 1000000) {
     int nsec = (x->tv_usec - y->tv_usec) / 1000000;
     y->tv_usec += 1000000 * nsec;
     y->tv_sec -= nsec;
  } 

  /* Compute the time remaining */
  /* tv_usec is certainly positive */
  result->tv_sec = x->tv_sec - y->tv_sec;
  result->tv_usec = x->tv_usec - y->tv_usec; 

 /* Return 1 if result is negative. */
  return x->tv_sec < y->tv_sec;
}

Condor and library paths

Fri, 30 Nov 2012 17:42:17 GMT

Sometimes a condor job will return an error beginning with:

condor_exec.exe: error while loading shared libraries

This can happen because the software you need is not installed on the execute node, in which case you should exclude that execute node in your submit file and let us know about the problem. You can safely assume that is the problem if you're still getting the error after following the steps outlined below.

More often, the software is on the execute node but is not in its default library search path. All third-party software is installed in /opt, which is a shared network filesystem common to all execute nodes as well as some other machines, including hooke. Thus if the program works in one place, it should work everywhere IF the default library paths are the same. Sometimes they aren't. We try to keep the nodes consistent, but there are over 100 of them in many different small clusters, and sometimes they diverge.

Fortunately, the problem can be easily remedied in the user's shell environment. The search path will add the paths in the environment variable LD_LIBRARY_PATH.

Here is an example using Gurobi 4.5:

condor_exec.exe: error while loading shared libraries: libgurobi45.so: cannot open shared object file: No such file or directory

The problem here is most likely that the execute node doesn't know where to find the shared library file it needs, libgurobi45.so. It should be able to, if wren could, but sometimes that doesn't happen even though the software is actually there.

Fortunately, this is a very easy fix. We only need to find the location of libgurobi45.so, and add that directory path to our environment on the execute node.

The library should be in a fairly obvious place. All special third-party applications are installed in /opt, and most libraries are in a "lib" or "lib64" subdirectory somewhere in their application's directory tree. If we do an "ls" in /opt we find that Gurobi 4.5 is in /opt/gurobi450. It's quickly obvious from there that the most likely library path is /opt/gurobi450/linux64/lib, and an "ls" on that directory does in fact show that it contains libgurobi45.so.

Alternatively, we could use the "find" command, like this:


wren:~ $ cd /opt/gurobi
wren:/opt/gurobi $ find . -name libgurobi45.so 2>/dev/null
./linux64/lib/libgurobi45.so

Once we know the directory that contains the needed library, we need to add it to the environment variable LD_LIBRARY_PATH on the execute node. This in can be done in two ways:

1. Add it to your login profile (usually via .profile, .bashrc, or .bash_profile) and ensure "getenv = True" is in your submit file so that the execute node uses your login environment. The syntax in your login profile is:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/gurobi450/linux64/lib

This is usually the best method.

2. If for some reason you don't want your entire login environment exported on the submit node, you could add the needed environment variable to the submit file explicitly:

Environment LD_LIBRARY_PATH=/opt/gurobi450/linux64/lib

Condor submit file examples

Fri, 30 Nov 2012 17:36:24 GMT

Some example submit file syntax all would go before the "queue" command.

Force a job to run on a single node:

+Requirements = Machine == "boyle16.isye.gatech.edu"

Allow each queued process to run on any one of a specified set of multiple nodes:

+Requirements = (Machine == "boyle1.isye.gatech.edu" || Machine == "boyle2.isye.gatech.edu" || Machine == "boyle3.isye.gatech.edu")

Thus far the only method we've gotten to work is the use of a separate full test statement for each desired machine. We've seen no examples using wildcard or syntax shortcuts, and the obvious guesses fail.

Also, if all specified nodes are unavailable, the job will remain idle with LastRejMatchReason = "no match found"

To disallow specific nodes, simply replace the operator:

+Requirements = (Machine != "goddard1.isye.gatech.edu" || Machine != "goddard2.isye.gatech.edu")

This can be useful if you find that software is misconfigured on a given set of nodes, etc. We try to avoid this but it does occasionally happen.

Dedicated Queue

Fri, 30 Nov 2012 17:31:41 GMT

Condor's primary queue is oriented towards cycle harvesting. This means that jobs running in the vanilla universe on Wren are subject to preemption for one of several reasons.

Some users need to run long jobs without checkpointing, where preemption means restarting from the beginning and losing days of computing time. As a practical matter it can often mean never finishing the job at all.

To alleviate this problem we have provided a dedicated queue. This queue runs from a different submission machine, hooke (hooke.isye.gatech.edu). The queue does have a disadvantage in that it has fewer assigned machines and you may have to wait longer for a slot to open up. However, if you have just a few long-running jobs, it's the way to go.

Submitting from hooke requires only a couple of changes.

First, obviously, you must log into hooke to submit the job. This can be done simply by doing "ssh hooke" whenever your code is ready to go. You don't need to do any of the development on hooke; that should be done on wren and then just log into hooke when you're ready to submit. Once you've submitted the job, you will need to monitor it on hooke as well. Wren's queue and history will not contain the details of the job.

Finally, the submit file requires a couple of changes. The universe should be changed to "parallel", and the line "machine count = 1" needs to be added before the"queue" statement.

So an example submit file for the dedicated queue from hooke looks like this:


universe   = parallel
executable = mycode
output     = mycode.out
error      = mycode.err
arguments  = 10 20
machine_count = 1
queue

Once you have this in place you can submit the job normally via "condor_submit", and the job should normally run to completion rather than being subject to preemption.

MPI Jobs

Fri, 30 Nov 2012 17:25:14 GMT

Condor supports high-speed distributed parallel computing using OpenMPI. MPI jobs require use of the dedicated queue on hooke, and can't be submitted from wren.

MPI jobs are run on the Boyle nodes, consisting of 16 eight-core machines with a low-latency Infiniband cross-connect between the nodes. OpenMPI jobs automatically use the fastest available transport method for message passing between processes.

Condor handles MPI jobs via a front end wrapper script called ompiscript (full path: /usr/local/bin/ompiscript). This wrapper is submitted as the executable name, and the submitter's MPI executable becomes the first command line argument. The wrapper automates all of the usual tasks associated with starting an MPI job: creates a hosts file, does the "mpirun" command, and takes care of passwordless ssh access.

Of course it remains crucial that users make sure their MPI code works in the shell before submitting it to Condor.

Submit file examples follow.

Example 1: running on any available nodes

The generic form of MPI submission will run on any available CPU (currently this means just on the boyle nodes). Contact IT first if you intend to submit an MPI job to more than 32 CPUs.

universe = parallel
executable = /usr/local/bin/ompiscript
arguments = exe1 argv1 argv2 ....
getenv = True          ## needed for your env to be present on execute nodes
output = exe1.out   ## stdout
error = exe.err    ## stderr
machine_count = 32
queue

If you want separate output files for each process, you can use a macro like this:

output = out/exe1.out.$(NODE)
error = out/exe1.err.$(NODE)

Example 2: Running a job on a specific node

If a job is run on 8 CPUs it can be kept on a single node, which will produce slightly faster interprocess communication. The user will need to find a node with no jobs currently running on it and specify that node in the command file.

universe = parallel
executable = /usr/local/bin/ompiscript
arguments = exe2 argv1 argv2 ....
getenv = True

output = exe2.out
error = exe2.err
machine_count = 8
queue

Running multiple iterations in condor

Thu, 06 Dec 2012 16:25:56 GMT

Many ISyE research problems involve running many separate iterations of the same code with either random or predetermined inputs. Condor excels at these types of problems.

When running on a single computer, the most common solution is set up loops and let just one or a few processes iterate through the inputs (sometimes generating random inputs along the way).

However, it is much more convenient and faster to let Condor handle the looping. If you can feed Condor multiple small jobs - anywhere from a minute to an hour in expected run time - it can spread them out onto all available resources and finish them very quickly.

Example 1: C program

We need to process 1000 different sets of input constraints using a compiled executable named "program0", written in C with the first argument being the input file.

First in our working directory we do 'mkdir in; mkdir out'. Put the data in 1000 separate files in the input directory, with names containing the integers 0 through 999. Here we will use 0.in through 999.in.

Then create the Condor submit file, 'program0.cmd':


universe    = vanilla
executable  = program0
arguments   = in/$(Process).in
output      = out/$(Process).out
error       = out/$(Process).err
queue 1000

Submit this script, and you should have 1000 jobs queued up that will iterate through your data and create 1000 output files.

Example 2 : Matlab

First, we need for our Matlab program to have an input variable for the process number, taken from Condor's $(Process) macro. Then we use that variable within the Matlab program to determine the name of the file to read.

Then the Condor submit file looks like this:


universe = vanilla
executable  = /usr/local/bin/matlab
arguments   = -singleCompThread -r clear -r procnum=$(Process)
input       = program0.m
output      = out/$(Process).out
error       = out/$(Process).err
queue 1000

Random inputs:

The above examples assume that the user has a predetermined set of input data to use, but what about generating the data at random in the program?

That can be done, but there is a major catch: the random seed needs to be the same to ensure valid randomness.

This could be accomplished by importing a constant random seed within the program. But the far easier method is to use a single program to generate the random input data you need, then set it up as in the examples above.

Outlook 2010 Setup Instruction for the New IMAP Accounts

Fri, 09 Sep 2011 18:06:51 GMT

Problem:

How do I set up my new IMAP email account on a new or existing installation of Outlook 2010?

Solution:

In Outlook 2010, click the File tab and click the Account Settings button.
It will display the Account Settings window with Email tab open.
Click New option to open the Add New Account Windows. Select Manually configure server settings or additional server types. Then click Next button.
Select Internet E-mail, then click Next .
- Enter your full name as "Your Name".
- Enter your Email Address (such as "allen.belletti@isye.gatech.edu".)
- Select IMAP as Account Type
- Set the Incoming mail server (IMAP) to "imap.isye.gatech.edu".
- Set the Outgoing mail server (SMTP) to "mail.isye.gatech.edu"
- Enter your ISyE account name for the User Name (for example, "allen".)
- Enter your ISyE account password.
- It is recommended that you uncheck "Remember password".
- Make sure that "Require logon using Secure Password Authentication (SPA)" is not checked.
Click "More Settings" tab to open the Internet Email Settings window.
- Click the "Outgoing Server" tab.
- Check "My outgoing server (SMTP) requires authentication."
- Ensure that "Use same settings as my incoming mail server" is selected.
Click Sent Items tab.
- Check "Save sent items in the following folder on the server" .
- Select Sent folder in the ISyE email list.
Click Deleted Items tab.
- Check "Move deleted items to the following folder on the server"
- Select Trash folder in the ISyE email folder list.
- Uncheck Purge items when switching folders while online
Click Advanced tab.
- Select port 143 for incoming server (IMAP) and 587 for Outgoing server (SMTP)
- Select TLS as the encrypted connection for both Incoming and Outgoing server.
Click OK to close Internet E-mail Settings window.
Click Next in Add New Account window, then click Finish to close the window.

Installing the GT Root certificate (fixes intranet errors)

Tue, 14 Jun 2011 17:24:58 GMT

The ISyE Intranet is now encrypted with a Georgia Tech SSL certificate. This means that "out of the box" most browsers will show an error message when you try to visit the Intranet.

Installing the GT certificate should prevent this problem. To install the GT certificate, visit ca.gatech.edu.

The web page there should automatically detect your browser and operating system, and give you the instructions to install the certificate.

NOTE: this will only work for the browser you are using at the time. If you use Firefox and Chrome, you may need to install it for each browser. If you have more that one computer, like a desktop and a laptop, you will need to install it on each system.

Copying files from your incoming ftp directory

Mon, 24 Jan 2011 19:55:52 GMT

If you have had files placed in an incoming ftp directory, here is the method for retrieving those files.

ssh to castle.isye.gatech.edu using SecureCRT or another ssh client
ssh from castle to ftp.isye.gatech.edu
copy the files using cp

See the transcript below as an example:

zero:~ lsharp$ ssh castle.isye.gatech.edu

Unauthorized system access is prohibited.

System use is governed by the GIT Computer and
Network Usage, Security and Data Access Policies.

Warning: untrusted X11 forwarding setup failed: xauth key data not generated
Warning: No xauth data; using fake authentication data for X11 forwarding.
Last login: Mon Jan 24 14:36:21 2011 from adsl-074-244-119-076.sip.asm.bellsouth.net

----[ 10/29/2010 ]---------------------------------------------------------

Castle is now enforcing limits on CPU time and memory for all users.  You
may run for up to 60 minutes of CPU time and use up to 4GB of memory.  If
either of these are exceeded, the job will automatically be killed by the
system.

This machine is intended for general use and development, so these limits
should not pose a problem for anyone and will ensure that the system
remains responsive for all.  Please contact us via helpdesk@isye.gatech.edu
if you have questions or concerns.  Long running and/or large jobs should
be run via the ISyE Compute infrastructure.  Please contact Trey Palmer
(trey@isye.gatech.edu) for additional information.

Also please note that keep.isye.gatech.edu is available to everyone as an
alternate to castle, should the latter become unavailable at any point.

--------------------------------------------------------------------------

-bash-3.2$

-bash-3.2$ ssh ftp.isye.gatech.edu

Unauthorized system access is prohibited.

System use is governed by the GIT Computer and
Network Usage, Security and Data Access Policies.


lsharp@ftp.isye.gatech.edu's password: 
Last login: Thu Oct 23 10:53:35 2008 from argo.isye.gatech.edu
-bash-3.2$

-bash-3.2$ cd /ftp/incoming
-bash-3.2$ cd lsharp
-bash-3.2$ ls -l
total 22020
-rw-r--r--+ 1 ftp ftp      131 Jan 12  2010 callhome.sh
-rw-r--r--+ 1 ftp ftp 24145004 Jan 12  2010 vlc-0.8.6b.dmg
-rw-r--r--+ 1 ftp ftp    42387 Sep  1 09:13 z5b9J.gif
-bash-3.2$ cp z5b9J.gif ~lsharp

Web Server Migration

Tue, 13 Apr 2010 20:28:53 GMT

ISyE Web Server Upgrade

ISyE is moving our web servers to new hardware in the coming weeks. We are using this opportunity to make some security fixes to the server software. For the majority of ISyE users this will have no impact on your web pages. A few users may have to make some changes to their code to work with the new security settings.

How Can I Know If My Web Page Will Work?

Our current web server uses the url http://www2.isye.gatech.edu. The new web server is using the url http://www3.isye.gatech.edu.

So if your current web pages are located at http://www2.isye.gatech.edu/workforcecom/ you can see them viewed on the new server by changing the url from www2 to www3: http://www3.isye.gatech.edu/workforcecom/.

There are two main groups of users who should check their code: Users who have web forms, and users who connect to databases.

Database users

In the past, the web server and the database server were both on the same hardware. Database connections to 'localhost' would work. The new web server is separate from the database server, so localhost connections will no longer work. If you have a page that connects to a database as localhost or ballista.isye.gatech.edu, you will need to change the connection to mysql.isye.gatech.edu

Web Form users

The old web server had the php setting Register Globals turned on. This meant that in web forms, each field you submitted would be converted to a php variable. For example, if you had a form that requested the field "firstname" it would be automatically converted to a php variable $firstname. On the new server this setting is turned off, so variables will not be automatically created.

If you have a web form that requests the field "firstname" now you will need to either address the POST variable directly, as $_POST['firstname'] or copy its value with the following line of code:

$firstname = $_POST['firstname'];

Having problems?

If you have any problems with your web pages or have any further questions about these changes, email us at helpdesk@isye.gatech.edu and we will be glad to help you.

Thunderbird 3 setup instructions

Tue, 05 Jan 2010 18:18:13 GMT

Problem:

How do I set up my new IMAP email account on a new or existing installation of Thunderbird 3.0?

Solution:

Configuring Thunderbird 3.0

New Installation

1. Download thunderbird from http://getthunderbird.com

2. Install using all the default setup options. At the end of the install you will be able to launch Thunderbird.

3. When Thunderbird starts, you will be presented with the Mail Account Setup window. Enter your name, your ISyE Email and your ISyE password.

4. Thunderbird should autodetect the settings for ISyE mail servers as follows:

5. The Incoming setup is correct, but the outgoing setup is not, so click on the Manual Setup in the lower left corner of the window

6. The Account Settings window will appear. Click the "Outgoing Server (SMTP)" option at the bottom of the list on the left

7. When the Outgoing Server settings appear, click the Edit button on the right side.

8. When the SMTP Server window appears, go to the "Connection Security" pulldown at the bottom and select "STARTTLS"

9. Click OK in the SMTP Server Window.

10. Click OK in the Account Settings window.

11.Click OK in the Mail Account Setup window.

Thunderbird should now successfully connect to the ISyE Mail servers and download your messages.

Install lightning and Create ICS calendar

Thu, 28 May 2009 18:11:04 GMT

Part I: Install Lightening Add-on in Thunderbird

Follow the instruction online by click the link below.  
http://www.mozilla.org/projects/calendar/lightning/download.html

Part II: Create ICS calendar in Thunderbird to sync with My Gatech Calendar

Open the calendar in Thunderbird.
Go to Calendar, click New Calendar to open the window.
Select on the Network and click Next
Select iCalendar (ICS). For the location, input https://mail.gatech.edu/home/@mail.gatech.edu/Calendar
( is your GT Unix account. ) Then click Next.
Give a name to the calendar you created. Leave E-mail as none. Click Next.
You may be asked for the user name and password for your GT account.
Click finish.
After the calendar is created, right click it to open its properties setting. Uncheck Read Only property.

Setting file permissions for web files

Mon, 02 Feb 2009 19:04:16 GMT

Files must be made world readable to be accessible on the web.

From the command line:

chmod a+r filename

From Fugu on a Mac:

Bring up the file info window by:

a) selecting the file and pressing Command-I on the keyboard, or
b) selecting the file and clicking the Info button at the top of the window, or
c) right clicking on the file and selecting "Get Info" from the menu

then in the permissions section, make sure the "Read" box is checked for the Owner, Group and Others, then click the Apply button.

From WinSCP on a PC:

Bring up the file info window by:

a) selecting the file and pressing F9 on the keyboard, or
b) selecting the file and clicking the Properties button at the bottom of the window, or
c) right clicking on the file and selecting "Properties" from the menu

then in the permissions section, make sure the "R" box is checked for the Owner, Group and Others, then click the OK button.

Directories need to be world readable and executable to be accessible on the web. The process is very similar to files:

From the command line:

chmod a+rx filename

From Fugu on a Mac:

Bring up the file info window by:

then in the permissions section, make sure the "Read" and "Execute" boxes are checked for the Owner, Group and Others, then click the Apply button.

From WinSCP on a PC:

Bring up the file info window by:

then in the permissions section, make sure the "R" and "X" boxes are checked for the Owner, Group and Others, then click the OK button.

Note PHD students only: Printing from a personal computer to ricphd

Fri, 27 Mar 2009 14:57:40 GMT

There is no way to directly print to that printer from a personal machine, however you can print by:

Using winscp, (or similar scp application) copy the files you want to print from your computer to one of the
ISyE unix computers, say castle.isye.gatech.edu

Then log into castle.isye.gatech.edu using SSH.

From the console type: lp -d ricphd "filename"

http://herald.isye.gatech.edu/helpdesk/index.php/kb/article/000009

IMAP Client Setup master instructions

Thu, 01 May 2008 19:23:48 GMT

Problem:

How do I configure my mail client to connect to ISyE's IMAP server?

Solution:

Introduction

This document provides step-by-step instructions to configure your email client for use with the ISyE IMAP email system. In order to use mail, you must first have an ISyE computer account. If you do not yet have an account, please visit http://www.isye.gatech.edu/apps/unix-account/ to apply for one.

The following clients are addressed herein:

Thunderbird 1.5/2.0

Evolution 2.0

Outlook 2003

Outlook 2000

Pine

Mutt

Eudora

In addition, if you are using a client other than the ones listed above, general connection information may be found at in the final section. Please note that the preferred client on all platforms is Thunderbird 2.0, and that the IT department suggests that you try using it first.

Please feel free to contact us with any questions or concerns via helpdesk@isye.gatech.edu.

Thunderbird 1.5/2.0 " new installation

From the "New Account Setup" window, select "Email account" and click "Next".
Enter your full name and preferred email address (for example, allen.belletti@isye.gatech.edu), click "Next".
Select server type "IMAP", enter "imap.isye.gatech.edu" as Incoming Server.
Enter "mail.isye.gatech.edu" as the Outgoing Server, click "Next".
Enter your ISyE account name (for example, "allen") as the Incoming User Name.
Enter the same account name as the Outgoing User Name, click "Next".
Choose a name for this account such as "ISyE Mail" and click "Next".
If everything is correct, click "Finish".
When prompted, enter your ISyE account password and click "OK".
Select Tools->Account Settings from the menu (note that on Linux, this will be Edit->Account Settings.)
Click on "Outgoing Server (SMTP)" at the bottom of the leftmost portion of the window.
Click the "Edit" button in the upper right, beneath "Outgoing Server (SMTP) Settings".
Click the button next to "TLS" (not "TLS, if available".)
Note: If you are at home and are unable to send mail, please try changing the Port number from "25" to "587" as some Internet Service Providers require this.
Click "OK".
Click "OK" again to exit the Account Settings page.

Thunderbird 1.5/2.0 " existing installation

Select Tools->Account Settings from the menu (note that on Linux, this will be Edit->Account Settings.)
Click the "Add Account" button in the bottom left.
Select "Email account" and click "Next".
Enter your full name and preferred email address (for example, allen.belletti@isye.gatech.edu), click "Next".
Select server type "IMAP", enter "imap.isye.gatech.edu" as Incoming Server, click "Next".
Enter your ISyE account name (for example, "allen") as the Incoming User Name, click "Next".
Choose a name for this account such as "ISyE Mail" and click "Next".
If everything is correct, click "Finish". The account name will appear in the left side of the window above items such as "Server Settings" and "Copies & Folders".
Click on "Outgoing Server (SMTP)" at the bottom of the leftmost portion of the window.
Click the "Add" button in the upper right, beneath "Outgoing Server (SMTP) Settings"
Enter "mail.isye.gatech.edu" for the Server Name.
Make sure that "Use name and password" is checked, and enter your account name next to "User Name".
Click the button next to "TLS" (not "TLS, if available".)
Note: If you are at home and are unable to send mail, please try changing the Port number from "25" to "587" as some Internet Service Providers require this.
Click "OK".
Click the name of your new account that appeared in the left-hand portion of the window (for example "ISyE Mail").
Near the bottom of the right-hand side you will see "Outgoing Server (SMTP)". Make sure that the "mail.isye.gatech.edu" entry that you created above has been selected.
Click on "Server Settings" located beneath the name of your new account, in the left-hand portion of the window.
Under "Security Settings" on the right side, select "TLS"
If desired, check the "Check for new messages at startup" button and also enter how often you wish to check for new messages (the default is every 10 minutes.)
Click "OK" and the "Account Settings" window will disappear. You are now ready to read and send email with Thunderbird. Note that you may need to click on "Inbox" to get started.

Evolution 2.0 " new installation

Start Evolution and click the "Forward" button.
Enter your Full Name (ie, Allen Belletti) and your preferred Email Address (such as "allen.belletti@isye.gatech.edu").
Leave "Make this my default account" checked and click "Forward".
Set the Server Type to "IMAP".
Enter "imap.isye.gatech.edu" for the Host.
Enter your ISyE account name (such as "allen") as the Username.
For Use Secure Connection (SSL), select "Always".
Leave Authentication Type set to "Password" and click "Forward".
If desired, check "Automatically check for new mail" and set an interval.
If you have a large number of folders, please uncheck "Check for new messages in all folders".
Click "Forward" to continue.
Confirm that the Server Type is "SMTP" and enter "mail.isye.gatech.edu" for the Host.
Check "Server requires authentication" and set Use Secure Connection (SSL) to "Whenever Possible".
Ensure that the Authentication Type is "PLAIN" and enter your ISyE account name as the Username. Click "Forward".
Choose a name for your account such as "ISyE Mail" and click "Forward".
Select the proper time zone and click "Forward".
Click "Apply"

Evolution 2.0 " existing installation

Select Tools->Settings from the menu.
Press the "Add" button in the upper-right of the window.
Click "Forward" to continue.
Enter your Full Name (ie, Allen Belletti) and your preferred Email Address (such as "allen.belletti@isye.gatech.edu").
Check "Make this my default account" and click "Forward".
Set the Server Type to "IMAP".
Enter "imap.isye.gatech.edu" for the Host.
Enter your ISyE account name (such as "allen") as the Username.
For Use Secure Connection (SSL), select "Always".
Leave Authentication Type set to "Password" and click "Forward".
If desired, check "Automatically check for new mail" and set an interval.
If you have a large number of folders, please uncheck "Check for new messages in all folders".
Click "Forward" to continue.
Confirm that the Server Type is "SMTP" and enter "mail.isye.gatech.edu" for the Host.
Check "Server requires authentication" and set Use Secure Connection (SSL) to "Whenever Possible".
Ensure that the Authentication Type is "PLAIN" and enter your ISyE account name as the Username. Click "Forward".
Choose a name for your account such as "ISyE Mail" and click "Forward".
Click "Apply"

Outlook 2003 " new installation

From the beginning of the Outlook 2003 Startup wizard:

Click "Next".
If asked about upgrading, choose "Do not upgrade" and click "Next".
When asked about configuring an email account, select "Yes" and click "Next".
Choose "IMAP" and click "Next".
Enter your full name as "Your Name".
Enter your preferred Email Address (such as "allen.belletti@isye.gatech.edu".
Set the Incoming mail server (IMAP) to "imap.isye.gatech.edu".
Set the Outgoing mail server (SMTP) to "mail.isye.gatech.edu"
Enter your ISyE account name for the User Name (for example, "allen".)
Enter your ISyE account password.
It is recommended that you uncheck "Remember password".
Make sure that "Log on using Secure Password Authentication (SPA)" is not checked.
Click the "More Settings" button.
Click the "Outgoing Server" tab.
Check "My outgoing server (SMTP) requires authentication."
Ensure that "Use same settings as my incoming mail server" is selected.
Click the "Advanced" tab.
Beneath "Incoming server (IMAP)", check the box for "This server requires an encrypted connection (SSL)". Do not check the second box beneath "Outgoing server (SMTP)".
Click "OK".
Click "Next".
Click "Finish".

Outlook 2003 " existing installation

Select Tools->Email Accounts from the menu.
Select "Add a new e-mail account" and click "Next".
Choose "IMAP" and click "Next".
Enter your full name as "Your Name".
Enter your preferred Email Address (such as "allen.belletti@isye.gatech.edu".
Set the Incoming mail server (IMAP) to "imap.isye.gatech.edu".
Set the Outgoing mail server (SMTP) to "mail.isye.gatech.edu"
Enter your ISyE account name for the User Name (for example, "allen".)
Enter your ISyE account password.
It is recommended that you uncheck "Remember password".
Make sure that "Log on using Secure Password Authentication (SPA)" is not checked.
Click the "More Settings" button.
Click the "Outgoing Server" tab.
Check "My outgoing server (SMTP) requires authentication."
Ensure that "Use same settings as my incoming mail server" is selected.
Click the "Advanced" tab.
Beneath "Incoming server (IMAP)", check the box for "This server requires an encrypted connection (SSL)". Do not check the second box beneath "Outgoing server (SMTP)".
Click "OK".
Click "Next".
Click "Finish".

Outlook 2000 " new installation

From the beginning of the Outlook 2000 Startup wizard

Click "Next"
If prompted to use settings from another version of Outlook, select "No" and click "Next".
If prompted to upgrade, select "None of the above" and click "Next".
Select "Internet Only" and click "Next".

If you are warned about not being able to use Exchange Server, click "Yes".
Enter your full name and click "Next".
Enter your preferred email address such as "allen.belletti@isye.gatech.edu" and click "Next".
For "My incoming mail server", select "IMAP".
Enter "imap.isye.gatech.edu" for "Incoming mail (POP3 or IMAP) server".
Enter "mail.isye.gatech.edu" for your "Outgoing mail (SMTP) server".
Click "Next" to continue.
Enter your ISyE account name for the "Account name" field.
Enter your ISyE account password.
It is recommended that you uncheck "Remember password".
Make sure that "Log on using Secure Password Authentication (SPA)" is not checked.
Click "Next" to continue.
Select "Connect using my local area network (LAN)" and click "Next".
Click "Finish".
Once Outlook is started, select Tools->Accounts from the menu.
Make sure that your new email account is selected and click "Properties".
Click the "Servers" tab.
Check the box next to "My server requires authentication".
Click the "Advanced" tab.
Check the box next to "This server requires a secure connection (SSL)" that is located beneath "Incoming mail (IMAP)". Do not check the one beneath "Outgoing mail (SMTP)".
Click the "IMAP" tab.
Enter "INBOX." (note the period) as the "Root folder path".
If you have a large number of folders, please un-check "Check for new messages in all folders" as this can cause a large load on the mail servers.

Click "Apply".
Select your ISyE inbox from the Folder list by clicking on it.
Select Tools->IMAP Folders from the menu.
Click the "Query" button which should retrieve a list of your folders.
Un-check the box next to "When displaying hierarchy in Outlook, show only subscribed folders."
From the View->Current View menu, check "Hide Messages Marked for Deletion."
Quit and restart Outlook.

Outlook 2000 " existing installation

Select Tools->Accounts from the menu.
Click "Add" and select "Mail".
Enter your full name and click "Next".
Enter your preferred email address such as "allen.belletti@isye.gatech.edu" and click "Next".
For "My incoming mail server", select "IMAP".
Enter "imap.isye.gatech.edu" for "Incoming mail (POP3 or IMAP) server".
Enter "mail.isye.gatech.edu" for your "Outgoing mail (SMTP) server".
Click "Next" to continue.
Enter your ISyE account name for the "Account name" field.
Enter your ISyE account password.
It is recommended that you uncheck "Remember password".
Make sure that "Log on using Secure Password Authentication (SPA)" is not checked.
Click "Next" to continue.
Select "Connect using my local area network (LAN)" and click "Next".
Click "Finish".
Make sure that your new email account is selected and click "Properties".
Click the "Servers" tab.
Check the box next to "My server requires authentication".
Click the "Advanced" tab.
Check the box next to "This server requires a secure connection (SSL)" that is located beneath "Incoming mail (IMAP)". Do not check the one beneath "Outgoing mail (SMTP)".
Click the "IMAP" tab.
Enter "INBOX." (note the period) as the "Root folder path".
If you have a large number of folders, please un-check "Check for new messages in all folders" as this can cause a large load on the mail servers.
Click "OK".
Click "Close".
Select your ISyE inbox from the Folder list by clicking on it.
Select Tools->IMAP Folders from the menu.
Click the "Query" button which should retrieve a list of your folders.
Un-check the box next to "When displaying hierarchy in Outlook, show only subscribed folders."
Click "OK".
From the View->Current View menu, check "Hide Messages Marked for Deletion."
Quit and restart Outlook.

Mac OS X Mail " existing installation

From the "Mail" menu , select "Preferences" and click the "Accounts" button in at the top of the Preferences window.
Click the "plus" icon at the bottom of the accounts window to add a new account.
In the "Add Account" window, enter your full name and preferred email address (for example, allen.belletti@isye.gatech.edu), along with the password for your ISyE email account click "Continue".
Select account type "IMAP", enter "imap.isye.gatech.edu" as Incoming Mail Server, and choose a name for this account, such as "ISyE IMAP" to enter into the "Description" field.
Enter your ISyE account name (for example, "allen") as the User Name. Click "Continue".
Click "Use Secure Sockets Layer (SSL)" in the "Incoming Mail Security" window. Select "Password" in the Authentication drop-down menu. Click "Continue".
Choose a name for this outgoing server such as "ISyE Mail" and enter it in the "Description" field. Click "Next".
Enter "mail.isye.gatech.edu" as the Outgoing Mail Server.
Click "Use Authentication" and enter the same account name and password, click "Continue".
If everything is correct, click "Take account online" and then click "Create".
When prompted, enter your ISyE account password and click "OK".
Note: If you are at home and are unable to send mail, please try changing the Port number from "25" to "587" as some Internet Service Providers require this.
To do this, go back to the Account Preferences window.
Click on your ISyE account and then select "Edit server list..." in the "Outgoing Mail Server (SMTP):" menu.
In the next window, click once on mail.isye.gatech.edu and click the "Advanced" button.
Change the "Server port:" field to 587 and click "OK".
Close the Preferences dialog.

Pine

For Pine users, we have found that setting the following options in .pinerc works well. This will cause Pine to access mail via IMAP from the new server cluster. Please note that this will work best from Castle. Feel free to contact us with any questions about using Pine on other hosts.

user-domain=isye.gatech.edu

smtp-server=mail.isye.gatech.edu

inbox-path={imap.isye.gatech.edu}INBOX

default-fcc=Sent

feature-list=enable-incoming-folders

folder-collections="IMAP Server" {imap.isye.gatech.edu}INBOX.[]

rsh-open-timeout=0

Mutt

The following options should be set to enable Mutt to access mail via IMAP. Please note that you must be using a version of Mutt which has been compiled using the proper options. The version on Castle is one such. Feel free to contact us with any questions about using Mutt on other hosts.

set folder=imaps://imap.isye.gatech.edu/INBOX

set spoolfile=imaps://imap.isye.gatech.edu/INBOX

set hostname=isye.gatech.edu

set use_domain

set imap_user=allen

Eudora

We do not currently recommend Eudora for use with the ISyE mail system. It is known to have difficulties with IMAP and further, is no longer supported by OIT. Consider using Thunderbird instead as it works well, is free, and is our departmental standard. Please feel free to contact us via helpdesk@isye.gatech.edu with any questions or concerns.

Importing the Georgia Tech root certificate on Windows

When using Microsoft Outlook 2003 you may receive a message that reads "The server you are connected to is using a security certificate that could not be verified. A certificate chain processed, but terminated in a root certificate which is not trusted by the trust provider. Do you want to continue using this server?" Follow these steps to install the correct certificate:

Browse to "http://ca.gatech.edu/root-formats.php".
At the very bottom of the page, right click where it says "Right click and download".
Select "Save Target As" or "Save Link As" and choose a location for the file. It should be named "gt-cert-chain.p7b".
If not already open, start Microsoft Internet Explorer.
Select Tools->Internet Options.
Click the "Content" tab.
Click the "Certificates" button.
Click the "Import" button.
Click "Next".
Enter the filename that you saved in step 3 or click "Browse" and locate that file. If using "Browse" you will need to select "PKCS #7 Certificates" next to "Files of type:" at the bottom of the dialog.
Click "Next".
Choose "Automatically select the certificate store based on the type of certificate" and click "Next".
Click "Finish".
You may be prompted about installing a certificate representing "Education and Research Client CA". If so, click "Yes".
Click "OK" in response to "The import was successful."

You may now close any open windows and quit Internet Explorer if you desire. Microsoft Outlook 2003 should no longer prompt you about an invalid certificate.

General Connection Information

If you are using a mail client which is not addressed in detail by this document, the following information should allow you to get connected:

Receiving mail: Server "imap.isye.gatech.edu", protocol IMAP, port 143 with TLS security. You may also use SSL on port 993 if necessary. Do not use "secure authentication" as this is not supported.

Sending mail: Server "mail.isye.gatech.edu", port 25. Authenticated login is preferred but not required. Mail may not be sent by unauthenticated users except when located in an ISyE building and connected directly to the network. Secure connection using TLS is preferred. Note: some Internet Service Providers will block this connection. If you find that you are unable to send mail, try using port 587 instead. This is an alternate which will often solve the problem.

START HERE: How to run Unix/Linux compute jobs using Condor

Mon, 25 Feb 2013 21:07:53 GMT

Instructions for the use of ISyE's computational Linux cluster:

Initial requirements

You need an ISyE Unix Account.

Logging in

The central 64-bit compute server is wren.isye.gatech.edu (or just "wren" if you are on ISyE's subnets).
You should be able to access wren via ssh using your usual ISyE Unix account and password. Your Unix home directory will be mounted and accessible.

Compute cluster hierarchy

The central server, Wren, is called the "head node". When compute jobs are ready to be run, the head node assigns execution of them to subordinate servers which are called "compute nodes".
Users only need to log into the head node. The compute nodes do not support direct logins, and they cannot mount home directories. Compute jobs must be submitted to them through the Condor batch queue scheduler.

Development

Development for 64-bit Linux compute jobs should be done on wren, and short interactive test runs can be performed there. You should test your program "locally" on the head node before submitting it to Condor. By "local" we mean executing the program in your login shell on wren.
You want to make sure your code starts and runs as you expect for at least a few minutes locally on wren, before you submit it to Condor. By testing the program locally first, you will save yourself and everyone a lot of headaches. Condor works great once your code is ready to go, but it's a very frustrating way to debug!
Long-running compute jobs and large batches of unattended short-running jobs are not allowed on the head node.
GUI development interfaces can be exported to the local X Windows desktop via ssh X forwarding (the ssh -X option).

Condor commands

Condor's executables are located in /opt/condor/bin, which should be in your default command path on wren, with man pages available for most important commands.

Generally the most useful Condor commands are:


   condor_submit -- submit jobs to the condor batch scheduler.

   condor_status -- show overall cluster status.

   condor_q      -- show current job queue and status of jobs.

   condor_rm     -- remove jobs from the condor queue.

Job submission

For most compute jobs, you will need to create a condor submit description file (a ".cmd" file) and then run "condor_submit .cmd".
An example .cmd file is listed below. Other examples can be found in /opt/condor/examples, and the syntax for .cmd files is documented in the condor_submit man page.
In the simplest cases, you can submit a job to condor by running "consub filename". This will work for a simple program that has no options, no input and output files (so if run locally, output would go to the screen). consub will create a simple filename.cmd file, copy the executable to a compute node and run the job, and then copy the output back to filename.out.

File accessibility on the compute nodes

You can run jobs directly from your Unix home directory tree, and all your files will be accessible to the compute nodes.
If you need more space than currently allocated, request it from IT.
It is possible to transfer files to the nodes, and then back to wren upon completion (should_transfer_files). On dedicated compute clusters like ours this is generally not the desired method. Your home directory is mounted on the compute nodes just as it is on wren or castle, so there is really no need to transfer files.

Log files

The Condor log file should be written to /tmp, rather than to your home directory tree. Any user can create a directory in /tmp. Typically just create /tmp/ and use that as an output path for the "log" directive (see example below).

Condor universe

You must specify a "universe" to run a condor job in. For basic compute jobs written in any language, add a line to use the "vanilla" universe. If you do not explicitly specify the universe, it will default to "standard" universe which will not work without special compile-time options.

Example .cmd file


    universe        = vanilla
    executable      = foo
    output          = foo.out
    error           = foo.err
    log             = /tmp//foo.log
    arguments       = 10 20 30
    queue

This command file should be called "foo.cmd" and placed in the same global scratch space subdirectory as the program "foo". You will then type "condor_submit foo.cmd" to run this compute job. Normal screen output will go to to "foo.out" and stderr will go to "foo.err". The program expects three command-line arguments which are "10", "20", and "30".

Note that the log file is placed in /tmp! This is very important! Log files written to NFS-mounted home or scratch directories will often break condor due to NFS locking problems. So, once again, create a directory for yourself in /tmp, and write your log files there.

Any other input and output files generated by the program will be accessed normally, as long as the path is accessible to the program (normally, this means in your home or scratch directory).

In many cases you may also need to run jobs that won't work correctly without your login environment variables (such as PATH, LD_LIBRARY_PATH, or software-specific environment variables). Since these won't be available by default on the compute nodes, you need to explicitly add them via the submit file using this line:


     getenv = true

Notification

The cluster will not normally notify you of job completion. If you'd like notification, you should add one of the following lines depending on the level of notification you would like:


    notification = error        
    notification = complete
    notification = always

The default email address for these notifications is the ISyE email address of the user submitting the jobs. If you'd like mail sent to an alternate address, add the following line:


    notify_user = my.email@domain.com

More on Condor

For further documentation, please see the Condor User's Manual.

How to securely delete sensitive data files.

Wed, 23 Apr 2008 21:48:43 GMT

Problem:

How to securely delete sensitive data files after looking at the spider generated log file.

Solution:

1. Check whether Eraser is installed on your computer.
This can be done by right clicking on the recycle bin. There should be 7 different ways of erasing your recycle bin.

If this option is not available you can either:

a.
Download then install the eraser application from the following location.
http://www2.isye.gatech.edu/intranet/modules/spider/EraserSetup32.exe
Note you will have be to be logged in with Administrative privileges to install this application.

b.
Send a request to helpdesk@isye.gatech.edu and we will install the application asap.

2. After ensuring that the eraser application is installed, delete any sensitive files like you normally would, thus sending the files to the recycle bin.

3. To securely erase them, right click on the recycle bin and select "Erase Recycle bin (1-pass Pseudorandom Data)"

You will get a prompt asking you to confirm erasing of recycle bin. Clicking "yes" will then securely empty your recycle bin.