Thursday, 13 December 2012

Stop STDOUT / NOHUP Build

#Use the below Command

nohup command >/dev/null 2>&1   # doesn't create nohup.out

Saturday, 17 November 2012

Thread Dump and Heap Dump

# Get the PID - Process Id using JPS

jps -v | grep server_name | awk '{ print $1 }'

# Take the Thread Dump

jstack PID >> thread_dumps.log

# Take the Heap Dump

jmap -dump:file=Heap_Dump.log PID


Simple JSTACK Script

#!/bin/bash
if [ $# -eq 0 ]; then
    echo >&2 "Usage: jstackSeries <pid> <run_user> [ <count> [ <delay> ] ]"
    echo >&2 "    Defaults: count = 10, delay = 0.5 (seconds)"
    exit 1
fi
pid=$1          # required
user=$2         # required
count=${3:-10}  # defaults to 10 times
delay=${4:-30} # defaults to 30 seconds
while [ $count -gt 0 ]
do
    sudo -u $user jstack -l $pid >jstack.$pid.$(date +%H%M%S.%N)
    sleep $delay
    let count--
    echo -n "."
done

Function to Convert BLOB Column

create or replace function F_blob_test(B BLOB)
return clob is
  c clob;
  n number;
begin
  if (b is null) then
    return null;
  end if;
  if (length(b)=0) then
    return empty_clob();
  end if;
  dbms_lob.createtemporary(c,true);
  n:=1;
  while (n+32767<=length(b)) loop
    dbms_lob.writeappend(c,32767,utl_raw.cast_to_varchar2(dbms_lob.substr(b,32767,n)));
    n:=n+32767;
  end loop;
  dbms_lob.writeappend(c,length(b)-n+1,utl_raw.cast_to_varchar2(dbms_lob.substr(b,length(b)-n+1,n)));
  return c;
end;

Useful UNIX Commands

# OS version and Details
uname -a

# Replace ^m
:%s/^V^M//g

# To Get the Delimited value
cut -f5 -d \~

#Mail Server Details
Look for relay Host in the below file.
more /etc/mail/sendmail.cf

# Find a word in
find . -exec grep hello {} \; -print

#To find a file in any directory, starting in your current directory
$ find . -name filename

#If you do not know the complete filename,use the wildcard \* at the start and end of the filename.
$ find . -name \*partialfilename\* -print

#Find only files (not directories) from your current directory. For example,
$ find . -type -f -print

#And to find only directory files,type
$ find . -type -d -print

#To find all files containing a particular string
$ find . -name "\*" -exec grep "string" {} \\; -print

#To find all files that are greater than "n" bytes and have not been accessed for more than "x" days,
$ find . -type f -size +nc -atime +x

For example, to find a file greater than 800 bytes and not accessed for more than 10 days, type
$ find . -type f -size +800c -atime +10
The c indicates that the file size is in bytes.


sftp -oPort=2222 user@123.456.789.012:/home/something


lftp -c 'open -e "set ftps:initial-prot ""; \
   set ftp:ssl-force true; \
   set ftp:ssl-protect-data true; "\
   <your FTP command>; " \
   -u "USERNAME","PASSWORD" \
   ftps://HOSTNAME:990 

JVM Arguments and Tuning

What is Heap space in Java?

 

When a Java program started Java Virtual Machine gets some memory from Operating System. Java Virtual Machine or JVM uses this memory for all its need and part of this memory is call java heap memory. Heap in Java generally located at bottom of address space and move upwards. whenever we create object using new operator or by any another means object is allocated memory from Heap and When object dies or garbage collected ,memory goes back to Heap space in Java, to learn more about garbage collection see how garbage collection works in Java.

 

How to increase size of Java Heap

Default size of Heap in Java is 128MB on most of 32 bit Sun's JVM but its highly varies from JVM to JVM  e.g. default maximum and start heap size for the 32-bit Solaris Operating System (SPARC Platform Edition) is -Xms=3670K and -Xmx=64M and Default values of heap size parameters on 64-bit systems have been increased up by approximately 30%. Also if you are using throughput garbage collector in Java 1.5 default maximum heap size of JVM would be Physical Memory/4 and  default initial heap size would be Physical Memory/16. Another way to find default heap size of JVM is to start an application with default heap parameters and monitor in using JConsole which is available on JDK 1.5 onwards, on VMSummary tab you will be able to see maximum heap size.By the way you can increase size of java heap space based on your application need and I always recommend this to avoid using default JVM heap values. if your application is large and lots of object created you can change size of heap space by using JVM command line options -Xms and -Xmx. Xms denotes starting size of Heap while Xmx denotes maximum size of Heap in Java. There is another parameter called -Xmn which denotes Size of new generation of Java Heap Space. Only thing is you can not change the size of Heap in Java dynamically, you can only provide Java Heap Size parameter while starting JVM.


Java Heap and Garbage Collection

As we know objects are created inside heap memory  and Garbage collection is a process which removes dead objects from Java Heap space and returns memory back to Heap in Java. For the sake of Garbage collection Heap is divided into three main regions named as New Generation, Old or Tenured Generation and Perm space. New Generation of Java Heap is part of Java Heap memory where newly created object allocated memory, during the course of application object created and died but those remain live they got moved to Old or Tenured Generation by Java Garbage collector thread on Major collection. Perm space of Java Heap is where JVM stores Meta data about classes and methods, String pool and Class level details. You can see How Garbage collection works in Java for more information on Heap in Java and Garbage collection.

OutOfMemoryError in Java Heap

When JVM starts JVM heap space is the initial size of Heap specified by -Xms parameter, as application progress objects creates and JVM expands Heap space in Java to accommodate new objects. JVM also run garbage collector to reclaim memory back from dead objects. JVM expands Heap in Java some where near to Maximum Heap Size specified by -Xmx and if there is no more memory left for creating new object in java heap , JVM throws  java.lang.outofmemoryerror and  your application dies. Before throwing OutOfMemoryError No Space in Java Heap, JVM tries to run garbage collector to free any available space but even after that not much space available on Heap in Java it results into OutOfMemoryError. To resolve this error you need to understand your application object profile i.e. what kind of object you are creating, which objects are taking how much memory etc. you can use profiler or heap analyzer to troubleshoot OutOfMemoryError in Java. "java.lang.OutOfMemoryError: Java heap space" error messages denotes that Java heap does not have sufficient space and cannot be expanded further while "java.lang.OutOfMemoryError: PermGen space" error message comes when the permanent generation of Java Heap is full, the application will fail to load a class or to allocate an interned string.

Java Heap dump

Java Heap dump is a snapshot of Java Heap Memory at a particular time. This is very useful to analyze or troubleshoot any memory leak in Java or any Java.lang.outofmemoryerror. There is tools available inside JDK which helps you to take heap dump and there are heap analyzer available tool which helps you to analyze java heap dump. You can use "jmap" command to get java heap dump, this will create heap dump file and then you can use "jhat - Java Heap Analysis Tool" to analyze those heap dumps.

10 Points about Java Heap Space

1. Java Heap Memory is part of Memory allocated to JVM by Operating System.

2. Whenever we create objects they are created inside Heap in Java.

3. Java Heap space is divided into three regions or generation for sake of garbage collection called New Generation, Old or tenured Generation or Perm Space.

4. You can increase or change size of Java Heap space by using JVM command line option -Xms, -Xmx and -Xmn. don't forget to add word "M" or "G" after specifying size to indicate Mega or Giga. for example you can set java heap size to 258MB by executing following command java -Xmx256m HelloWord.

5. You can use either JConsole or Runtime.maxMemory(), Runtime.totalMemory(), Runtime.freeMemory() to query about Heap size programmatic in Java.

6. You can use command "jmap" to take Heap dump in Java and "jhat" to analyze that heap dump.

7. Java Heap space is different than Stack which is used to store call hierarchy and local variables.

8. Java Garbage collector is responsible for reclaiming memory from dead object and returning to Java Heap space.

9. Don’t panic when you get java.lang.outofmemoryerror, sometimes its just matter of increasing heap size but if it’s recurrent then look for memory leak in Java.


The allocation of memory for the JVM is specified using -X options when starting Resin (the exact options may depend upon the JVM that you are using, the examples here are for the Sun JVM).
JVM option passed to Resin
Meaning
-Xms
initial java heap size
-Xmx
maximum java heap size
-Xmn
the size of the heap for the young generation


Resin startup with heap memory options
unix> bin/httpd.sh -Xmn100M -Xms500M -Xmx500M
win> bin/httpd.exe -Xmn100M -Xms500M -Xmx500M
install win service> bin/httpd.exe -Xmn100M -Xms500M -Xmx500M -install

It is good practice with server-side Java applications like Resin to set the minimum -Xms and maximum -Xmx heap sizes to the same value.
For efficient garbage collection, the -Xmn value should be lower than the -Xmx value.

Heap size does not determine the amount of memory your process uses

If you monitor your java process with an OS tool like top or taskmanager, you may see the amount of memory you use exceed the amount you have specified for -Xmx. -Xmx limits the java heap size, java will allocate memory for other things, including a stack for each thread. It is not unusual for the total memory consumption of the VM to exceed the value of -Xmx.

(thanks to Rob Lockstone for his comments)
There are essentially two GC threads running. One is a very lightweight thread which does "little" collections primarily on the Eden (a.k.a. Young) generation of the heap. The other is the Full GC thread which traverses the entire heap when there is not enough memory left to allocate space for objects which get promoted from the Eden to the older generation(s).
If there is a memory leak or inadequate heap allocated, eventually the older generation will start to run out of room causing the Full GC thread to run (nearly) continuously. Since this process "stops the world", Resin won't be able to respond to requests and they'll start to back up.
The amount allocated for the Eden generation is the value specified with -Xmn. The amount allocated for the older generation is the value of -Xmx minus the -Xmn. Generally, you don't want the Eden to be too big or it will take too long for the GC to look through it for space that can be reclaimed.

Each thread in the VM get's a stack. The stack size will limit the number of threads that you can have, too big of a stack size and you will run out of memory as each thread is allocated more memory than it needs.
The Resin startup scripts (httpd.exe on Windows, wrapper.pl on Unix) will set the stack size to 2048k, unless it is specified explicity. 2048k is an appropriate value for most situations.
JVM option passed to Resin
Meaning
-Xss
the stack size for each thread

-Xss determines the size of the stack: -Xss1024k. If the stack space is too small, eventually you will see an exception class java.lang.StackOverflowError .
Some people have reported that it is necessary to change stack size settings at the OS level for Linux. A call to ulimit may be necessary, and is usually done with a command in /etc/profile:
Limit thread stack size on Linux
ulimit -s 2048





Categories of Java HotSpot VM Options
 
Standard options recognized by the Java HotSpot VM are described on the Java Application Launcher reference pages for Windows, Solaris and Linux. This document deals exclusively with non-standard options recognized by the Java HotSpot VM:
·         Options that begin with -X are non-standard (not guaranteed to be supported on all VM implementations), and are subject to change without notice in subsequent releases of the JDK.
·         Options that are specified with -XX are not stable and are not recommended for casual use. These options are subject to change without notice.
 
Some Useful -XX Options
 
Default values are listed for Java SE 6 for Solaris Sparc with -server. Some options may vary per architecture/OS/JVM version. Platforms with a differing default value are listed in the description.
·         Boolean options are turned on with -XX:+<option> and turned off with -XX:-<option>.
·         Numeric options are set with -XX:<option>=<number>. Numbers can include 'm' or 'M' for megabytes, 'k' or 'K' for kilobytes, and 'g' or 'G' for gigabytes (for example, 32k is the same as 32768).
·         String options are set with -XX:<option>=<string>, are usually used to specify a file, a path, or a list of commands
Flags marked as manageable are dynamically writeable through the JDK management interface (com.sun.management.HotSpotDiagnosticMXBean API) and also through JConsole.The options below are loosely grouped into three categories.
·         Behavioral options change the basic behavior of the VM.
·         Performance tuning options are knobs which can be used to tune VM performance.
·         Debugging options generally enable tracing, printing, or output of VM information.


Behavioral Options

Option and Default Value
Description
-XX:-AllowUserSignalHandlers
Do not complain if the application installs signal handlers. (Relevant to Solaris and Linux only.)
-XX:AltStackSize=16384
Alternate signal stack size (in Kbytes). (Relevant to Solaris only, removed from 5.0.)
-XX:-DisableExplicitGC
Disable calls to System.gc(), JVM still performs garbage collection when necessary.
-XX:+FailOverToOldVerifier
Fail over to old verifier when the new type checker fails. (Introduced in 6.)
-XX:+HandlePromotionFailure
The youngest generation collection does not require a guarantee of full promotion of all live objects. (Introduced in 1.4.2 update 11) [5.0 and earlier: false.]
-XX:+MaxFDLimit
Bump the number of file descriptors to max. (Relevant  to Solaris only.)
-XX:PreBlockSpin=10
Spin count variable for use with -XX:+UseSpinning. Controls the maximum spin iterations allowed before entering operating system thread synchronization code. (Introduced in 1.4.2.)
-XX:-RelaxAccessControlCheck
Relax the access control checks in the verifier. (Introduced in 6.)
-XX:+ScavengeBeforeFullGC
Do young generation GC prior to a full GC. (Introduced in 1.4.1.)
-XX:+UseAltSigs
Use alternate signals instead of SIGUSR1 and SIGUSR2 for VM internal signals. (Introduced in 1.3.1 update 9, 1.4.1. Relevant to Solaris only.)
-XX:+UseBoundThreads
Bind user level threads to kernel threads. (Relevant to Solaris only.)
-XX:-UseConcMarkSweepGC
Use concurrent mark-sweep collection for the old generation. (Introduced in 1.4.1)
-XX:+UseGCOverheadLimit
Use a policy that limits the proportion of the VM's time that is spent in GC before an OutOfMemory error is thrown. (Introduced in 6.)
-XX:+UseLWPSynchronization
Use LWP-based instead of thread based synchronization. (Introduced in 1.4.0. Relevant to Solaris only.)
-XX:-UseParallelGC
Use parallel garbage collection for scavenges. (Introduced in 1.4.1)
-XX:-UseParallelOldGC
Use parallel garbage collection for the full collections. Enabling this option automatically sets -XX:+UseParallelGC. (Introduced in 5.0 update 6.)
-XX:-UseSerialGC
Use serial garbage collection. (Introduced in 5.0.)
-XX:-UseSpinning
Enable naive spinning on Java monitor before entering operating system thread synchronizaton code. (Relevant to 1.4.2 and 5.0 only.) [1.4.2, multi-processor Windows platforms: true]
-XX:+UseTLAB
Use thread-local object allocation (Introduced in 1.4.0, known as UseTLE prior to that.) [1.4.2 and earlier, x86 or with -client: false]
-XX:+UseSplitVerifier
Use the new type checker with StackMapTable attributes. (Introduced in 5.0.)[5.0: false]
-XX:+UseThreadPriorities
Use native thread priorities.
-XX:+UseVMInterruptibleIO
Thread interrupt before or with EINTR for I/O operations results in OS_INTRPT. (Introduced in 6. Relevant to Solaris only.)



Performance Options

Option and Default Value
Description
-XX:+AggressiveOpts
Turn on point performance compiler optimizations that are expected to be default in upcoming releases. (Introduced in 5.0 update 6.)
-XX:CompileThreshold=10000
Number of method invocations/branches before compiling [-client: 1,500]
-XX:LargePageSizeInBytes=4m
Sets the large page size used for the Java heap. (Introduced in 1.4.0 update 1.) [amd64: 2m.]
-XX:MaxHeapFreeRatio=70
Maximum percentage of heap free after GC to avoid shrinking.
-XX:MaxNewSize=size
Maximum size of new generation (in bytes). Since 1.4, MaxNewSize is computed as a function of NewRatio. [1.3.1 Sparc: 32m; 1.3.1 x86: 2.5m.]
-XX:MaxPermSize=64m
Size of the Permanent Generation.  [5.0 and newer: 64 bit VMs are scaled 30% larger; 1.4 amd64: 96m; 1.3.1 -client: 32m.]
-XX:MinHeapFreeRatio=40
Minimum percentage of heap free after GC to avoid expansion.
-XX:NewRatio=2
Ratio of new/old generation sizes. [Sparc -client: 8; x86 -server: 8; x86 -client: 12.]-client: 4 (1.3) 8 (1.3.1+), x86: 12]
-XX:NewSize=2.125m
Default size of new generation (in bytes) [5.0 and newer: 64 bit VMs are scaled 30% larger; x86: 1m; x86, 5.0 and older: 640k]
-XX:ReservedCodeCacheSize=32m
Reserved code cache size (in bytes) - maximum code cache size. [Solaris 64-bit, amd64, and -server x86: 48m; in 1.5.0_06 and earlier, Solaris 64-bit and and64: 1024m.]
-XX:SurvivorRatio=8
Ratio of eden/survivor space size [Solaris amd64: 6; Sparc in 1.3.1: 25; other Solaris platforms in 5.0 and earlier: 32]
-XX:TargetSurvivorRatio=50
Desired percentage of survivor space used after scavenge.
-XX:ThreadStackSize=512
Thread Stack Size (in Kbytes). (0 means use default stack size) [Sparc: 512; Solaris x86: 320 (was 256 prior in 5.0 and earlier); Sparc 64 bit: 1024; Linux amd64: 1024 (was 0 in 5.0 and earlier); all others 0.]
-XX:+UseBiasedLocking
Enable biased locking. For more details, see this tuning example. (Introduced in 5.0 update 6.) [5.0: false]
-XX:+UseFastAccessorMethods
Use optimized versions of Get<Primitive>Field.
-XX:-UseISM
Use Intimate Shared Memory. [Not accepted for non-Solaris platforms.] For details, see Intimate Shared Memory.
-XX:+UseLargePages
Use large page memory. (Introduced in 5.0 update 5.) For details, see Java Support for Large Memory Pages.
-XX:+UseMPSS
Use Multiple Page Size Support w/4mb pages for the heap. Do not use with ISM as this replaces the need for ISM. (Introduced in 1.4.0 update 1, Relevant to Solaris 9 and newer.) [1.4.1 and earlier: false]
-XX:+UseStringCache
Enables caching of commonly allocated strings.
 
-XX:AllocatePrefetchLines=1
Number of cache lines to load after the last object allocation using prefetch instructions generated in JIT compiled code. Default values are 1 if the last allocated object was an instance and 3 if it was an array.
 
-XX:AllocatePrefetchStyle=1
Generated code style for prefetch instructions.
0 - no prefetch instructions are generate*d*,
1 - execute prefetch instructions after each allocation,
2 - use TLAB allocation watermark pointer to gate when prefetch instructions are executed.
 
-XX:+UseCompressedStrings
Use a byte[] for Strings which can be represented as pure ASCII. (Introduced in Java 6 Update 21 Performance Release)
 
-XX:+OptimizeStringConcat
Optimize String concatenation operations where possible. (Introduced in Java 6 Update 20)
 


Debugging Options

Option and Default Value
Description
-XX:-CITime
Prints time spent in JIT Compiler. (Introduced in 1.4.0.)
-XX:ErrorFile=./hs_err_pid<pid>.log
If an error occurs, save the error data to this file. (Introduced in 6.)
-XX:-ExtendedDTraceProbes
Enable performance-impacting dtrace probes. (Introduced in 6. Relevant to Solaris only.)
-XX:HeapDumpPath=./java_pid<pid>.hprof
Path to directory or filename for heap dump. Manageable. (Introduced in 1.4.2 update 12, 5.0 update 7.)
-XX:-HeapDumpOnOutOfMemoryError
Dump heap to file when java.lang.OutOfMemoryError is thrown. Manageable. (Introduced in 1.4.2 update 12, 5.0 update 7.)
-XX:OnError="<cmd args>;<cmd args>"
Run user-defined commands on fatal error. (Introduced in 1.4.2 update 9.)
-XX:OnOutOfMemoryError="<cmd args>;
<cmd args>"
Run user-defined commands when an OutOfMemoryError is first thrown. (Introduced in 1.4.2 update 12, 6)
-XX:-PrintClassHistogram
Print a histogram of class instances on Ctrl-Break. Manageable. (Introduced in 1.4.2.) The jmap -histo command provides equivalent functionality.
-XX:-PrintConcurrentLocks
Print java.util.concurrent locks in Ctrl-Break thread dump. Manageable. (Introduced in 6.) The jstack -l command provides equivalent functionality.
-XX:-PrintCommandLineFlags
Print flags that appeared on the command line. (Introduced in 5.0.)
-XX:-PrintCompilation
Print message when a method is compiled.
-XX:-PrintGC
Print messages at garbage collection. Manageable.
-XX:-PrintGCDetails
Print more details at garbage collection. Manageable. (Introduced in 1.4.0.)
-XX:-PrintGCTimeStamps
Print timestamps at garbage collection. Manageable (Introduced in 1.4.0.)
-XX:-PrintTenuringDistribution
Print tenuring age information.
-XX:-TraceClassLoading
Trace loading of classes.
-XX:-TraceClassLoadingPreorder
Trace all classes loaded in order referenced (not loaded). (Introduced in 1.4.2.)
-XX:-TraceClassResolution
Trace constant pool resolutions. (Introduced in 1.4.2.)
-XX:-TraceClassUnloading
Trace unloading of classes.
-XX:-TraceLoaderConstraints
Trace recording of loader constraints. (Introduced in 6.)
-XX:+PerfSaveDataToFile
Saves jvmstat binary data on exit.
-XX:ParallelGCThreads=
Sets the number of garbage collection threads in the young and old parallel garbage collectors. The default value varies with the platform on which the JVM is running.
-XX:+UseCompressedOops
Enables the use of compressed pointers (object references represented as 32 bit offsets instead of 64-bit pointers) for optimized 64-bit performance with Java heap sizes less than 32gb.
-XX:+AlwaysPreTouch
Pre-touch the Java heap during JVM initialization. Every page of the heap is thus demand-zeroed during initialization rather than incrementally during application execution.
-XX:AllocatePrefetchDistance=
Sets the prefetch distance for object allocation. Memory about to be written with the value of new objects is prefetched into cache at this distance (in bytes) beyond the address of the last allocated object. Each Java thread has its own allocation point. The default value varies with the platform on which the JVM is running.
-XX:InlineSmallCode=
Inline a previously compiled method only if its generated native code size is less than this. The default value varies with the platform on which the JVM is running.
-XX:MaxInlineSize=35
Maximum bytecode size of a method to be inlined.
-XX:FreqInlineSize=
Maximum bytecode size of a frequently executed method to be inlined. The default value varies with the platform on which the JVM is running.
-XX:LoopUnrollLimit=
Unroll loop bodies with server compiler intermediate representation node count less than this value. The limit used by the server compiler is a function of this value, not the actual value. The default value varies with the platform on which the JVM is running.
-XX:InitialTenuringThreshold=7
Sets the initial tenuring threshold for use in adaptive GC sizing in the parallel young collector. The tenuring threshold is the number of times an object survives a young collection before being promoted to the old, or tenured, generation.
-XX:MaxTenuringThreshold=
Sets the maximum tenuring threshold for use in adaptive GC sizing. The current largest value is 15. The default value is 15 for the parallel collector and is 4 for CMS.