Monday, September 14, 2020

Bash scripting and shell programming linux part 4

Data manipulations and text transformations with SED:

  • SED=Stream Editor
  • A stream is data that travels from one process to another through pipe, one file to another as redirect or one device to another.
  • Sed performs text transformation on streams.
    • Examples:
      • substitute some text with other text
      • removes lines
      • Append text after given lines
      • Insert text before certain lines

 type -a sed

sed is /usr/bin/sed

sed is /bin/sed

Syntax:
sed 's/search-pattern/replacement-pattern/flags' file_name

Example:
echo 'I love my wife.' > love.txt
cat love.txt
I love my wife.
sed 's/my wife/sed/' love.txt
I love sed.

Scripting is case sensitive so we can use use "i" flag to make it  case insensitive with sed
sed 's/MY WIFE/sed/I' love.txt
or
sed 's/MY WIFE/sed/i' love.txt

to append a line to a existing a file:
echo 'this is line 2.' >> love.txt
cat love.txt
I love my wife.
this is line 2.

Change my wife with sed
cat love.txt
I love my wife.
this is line 2.
I love my wife with all of my heart.
sed 's/my wife/sed/' love.txt
I love sed.
this is line 2.
I love sed with all of my heart.


sed 's/my wife/sed/' love.txt
I love sed.
this is line 2.
I love sed with all of my heart.
I love sed and my wife loves me too

globally change all the words "my wife" with sed:
sed 's/my wife/sed/g' love.txt
I love sed.
this is line 2.
I love sed with all of my heart.
I love sed and sed loves me too

change the 2nd word "my wife" in a line
sed 's/my wife/sed/2' love.txt
I love my wife.
this is line 2.
I love my wife with all of my heart.
I love my wife and sed loves me too

Save in a file
sed 's/my wife/sed/2' love.txt >ex.txt
cat ex.txt
I love my wife.
this is line 2.
I love my wife with all of my heart.
I love my wife and sed loves me too

delete a line starting with "this"
sed '/this/d' love.txt
I love my wife.
I love my wife with all of my heart.
I love my wife and my wife loves me too

using escape characters:
echo '/home/test' | sed 's/\/home\/test/\/home\/var\/test/'
/home/var/test

echo '/home/test' | sed 's:\/home\/test:\/home\/var\/test:'
/home/var/test

To delete a line based on a word/char starting with the line.
echo '#User to run service as.' > config
echo 'User apache' >> config
echo '#Group to run service as ' >> config
echo 'Group apache' >>config
 cat config
#User to run service as.
User apache
#Group to run service as
Group apache
sed '/^#/d' config
User apache
Group apache

awk:

  • Awk is a scripting language used for manipulating data and generating reports.The awk command programming language requires no compiling, and allows the user to use variables, numeric functions, string functions, and logical operators.
  • Awk is a utility that enables a programmer to write tiny but effective programs in the form of statements that define text patterns that are to be searched for in each line of a document and the action that is to be taken when a match is found within a line. Awk is mostly used for pattern scanning and processing. It searches one or more files to see if they contain lines that matches with the specified patterns and then performs the associated actions.
  • Awk is abbreviated from the names of the developers – Aho, Weinberger, and Kernighan.
Syntax:

awk options 'selection _criteria {action }' input-file > output-file

Options:
-f program-file : Reads the AWK program source from the file 
                  program-file, instead of from the 
                  first command line argument.
-F fs : Use fs for the input field separator


To print every lines of a file:
awk '{print}' config
#User to run service as.
User apache
#Group to run service as
Group apache

Print the lines which matches with the given pattern.
awk '/User/ {print}' config
#User to run service as.
User apache

Splitting a Line Into Fields :
For each record i.e line, the awk command splits the record delimited by whitespace character by default and stores it in the $n variables. If the line has 4 words, it will be stored in $1, $2, $3 and $4 respectively. Also, $0 represents the whole line.

awk '{print $1, $3}' config
#User run
User
#Group run
Group

Built In Variables In Awk:
Awk’s built-in variables include the field variables—$1, $2, $3, and so on ($0 is the entire line) — that break a line of text into individual words or pieces called fields.

NR: NR command keeps a current count of the number of input records. Remember that records are usually lines. Awk command performs the pattern/action statements once for each record in a file.

NF: NF command keeps a count of the number of fields within the current input record.

FS: FS command contains the field separator character which is used to divide fields on the input line. The default is “white space”, meaning space and tab characters. FS can be reassigned to another character (typically in BEGIN) to change the field separator.

RS: RS command stores the current record separator character. Since, by default, an input line is the input record, the default record separator character is a newline.

OFS: OFS command stores the output field separator, which separates the fields when Awk prints them. The default is a blank space. Whenever print has several parameters separated with commas, it will print the value of OFS in between each parameter.

ORS: ORS command stores the output record separator, which separates the output lines when Awk prints them. The default is a newline character. print automatically outputs the contents of ORS at the end of whatever it is given to print.


awk '{print NR,$1}' config
1 #User
2 User
3 #Group
4 Group

 awk '{print NR,$0}' config
1 #User to run service as.
2 User apache
3 #Group to run service as
4 Group apache

awk '{print NR,NF,$0}' config
1 5 #User to run service as.
2 2 User apache
3 5 #Group to run service as
4 2 Group apache

To print the first item along with the row number(NR) separated with ” – “
awk '{print NR "-" $1}' config
1-#User
2-User
3-#Group
4-Group

To print any non empty line if present
awk 'NF > 0' config
#User to run service as.
User apache
#Group to run service as
Group apache

 To find the length of the longest line present in the file:
 awk '{ if (length($0) > max) max = length($0) } END { print max }' config

25

To count the lines in a file:
awk 'END {print NR} ' config
4

Printing lines with more than 10 characters:
 awk 'length($0) > 10' config
#User to run service as.
User apache
#Group to run service as
Group apache

To print the squares of first numbers from 1 to n say 6:
awk 'BEGIN { for(i=1;i<=6;i++) print "square of", i, "is",i*i; }'
square of 1 is 1
square of 2 is 4
square of 3 is 9
square of 4 is 16
square of 5 is 25
square of 6 is 36




Friday, September 11, 2020

Curl command - Test a rest API

About curl command:

  • curl is a command line tool to transfer data to or from a server, using any of the supported protocols (HTTP, FTP, IMAP, POP3, SCP, SFTP, SMTP, TFTP, TELNET, LDAP or FILE).
  •  curl is powered by Libcurl. This tool is preferred for automation, since it is designed to work without user interaction. curl can transfer multiple file at once

syntax:

curl [options] [URL...]

Here are the options that we’ll use when making requests:

-X, --request - The HTTP method to be used.

-i, --include - Include the response headers.

-d, --data - The data to be sent.

-H, --header - Additional header to be sent.


Verbose:

When we are testing, it’s a good idea to set the verbose mode on:

curl -v http://www.example.com/

As a result, the commands would provide helpful information such as the resolved IP address, the port we are trying to connect to and the headers.

Example:

curl  -v -d '{"Customer": {"ID": "ID10","Name": "Name11","Address": "Address12"}}' -H 'Content-Type: application/json' http://localhost:8001/soa-infra/resources/POC/LoopInBPELProject/RestService/abc

* Connected to host  (host ip) port 8001

> POST /soa-infra/resources/POC/LoopInBPELProject/RestService/abc HTTP/1.1

User-Agent: curl/7.9.3 (powerpc-ibm-aix5.3.0.0) libcurl 7.9.3 (OpenSSL 0.9.8y) (ipv6 enabled)

Host: localhost:8001

Pragma: no-cache

Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*

Content-Type: application/json

Content-Length: 68

{"Customer": {"ID": "ID10","Name": "Name11","Address": "Address12"}}

<CustomersRes xmlns="http://www.cust.abc">

   <Outcome>ID10</Outcome>

</CustomersRes>

* Connection #0 left intact

* Closing connection #0

Output:

By default, curl outputs the response body to standard output. Optionally, we can provide the output option to save to a file:

curl -o out.json http://www.example.com/index.html

This is especially helpful when the response size is large.

Example:

curl  -o response.xml -d '{"Customer": {"ID": "ID10","Name": "Name11","Address": "Address12"}}' -H 'Content-Type: application/json' http://host:8001/soa-infra/resources/POC/LoopInBPELProject/RestService/abc
-rw-r--r--    1 oracle   dba              86 Sep 11 06:20 response.xml
cat response.xml
<CustomersRes xmlns="http://www.cust.abc">
   <Outcome>ID10</Outcome>
</CustomersRes>

POST:
We use this method to send data to a receiving service. And for that, we use the data option.

curl -d 'id=9&name=sri' http://localhost:8082/testurl
or, pass a file containing the request body to the data option like this:

curl -d @request.json -H "Content-Type: application/json" 
  http://localhost:8082/testurl

Example:
curl   -d @request.json -H 'Content-Type: application/json' http://host:8001/soa-infra/resources/POC/LoopInBPELProject/RestService/abc
<CustomersRes xmlns="http://www.cust.abc">
   <Outcome>ID10</Outcome>
</CustomersRes>

GET:
This is the default method when making HTTP calls with curl. In fact, the examples previously shown were plain GET calls.
While running a local instance of a service at port 8082, we'd use something like this command to make a GET call:
curl -v http://localhost:8082/testurl

Authentication:
If the API endpoint requires authentication, you’ll need to obtain an access key. Otherwise, the API server will respond with the “Access Forbidden” or “Unauthorized” response message.

The process of obtaining an access key depends on the API you’re using. Once you have your access token you can send it in the header:

curl -X GET -H "Authorization: Bearer {ACCESS_TOKEN}" "https://api.server.io/posts"

-u  option: curl also provides options to download files from user authenticated FTP servers.
Syntax:

curl -u {username}:{password} [FTP_URL]


how to store the rest response to a variable using shell script:
cat > curl.sh
#!/usr/bin/bash
var=$(curl -d '{"Customer": {"ID": "ID10","Name": "Name11","Address": "Address12"}}' -H 'Content-Type: application/json' http://uadcc-elmweb01:8001/soa-infra/resources/POC/LoopInBPELProject/RestService/abc)
echo $var

chmod 777 curl.sh
./curl.sh
  % Total    % Received % Xferd  Average Speed          Time             Curr.
                                 Dload  Upload Total    Current  Left    Speed
100    86  100    86    0     0   1954      0  0:00:00  0:00:00  0:00:00 86000
<CustomersRes xmlns="http://www.cust.abc"> <Outcome>ID10</Outcome> </CustomersRes>

12c SOA - How to call a shell script from BPEL using java embedding

 Implementation steps:

Step 1: Create a SOA project with empty composite.

Step 2: Create an XSD.

<?xml version="1.0" encoding="windows-1252" ?>

<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns="http://www.example.org"

            targetNamespace="http://www.example.org" elementFormDefault="qualified">

  <xsd:element name="Input">

    <xsd:complexType>

      <xsd:sequence>

        <xsd:element name="num1" type="xsd:string"/>

        <xsd:element name="num2" type="xsd:string"/>

      </xsd:sequence>

    </xsd:complexType>

  </xsd:element>

    <xsd:element name="Output">

    <xsd:complexType>

      <xsd:sequence>

        <xsd:element name="result" type="xsd:string"/>

      </xsd:sequence>

    </xsd:complexType>

  </xsd:element>

</xsd:schema>


Step 3: Create a Synchronous BPEL service using that schema exposing as a web service.

Step 4: Create a Shell script file and saved in UNIX directory.

Test.sh:

#!/usr/bin/bash

num1=$1

num2=$2

function add(){

sum=`expr $num1 + $num2`

echo "$sum"

return $sum

}

add $num1 $num2


Step 5 : Create a Java class(Right click on project⇾ New Galary⇾ Class(Java))

Test.java:

package invokeshellscriptproject;

import java.io.*;

public class Test {

          public String ShellScript(int i, int j) throws IOException

          {

               String Var=null;

          try{

                Process P;

                System.out.println(i);

                System.out.println(j);

              String strParam = "bash /soashare/shellscript/test.sh "

                                                     +i+" "+j+" ";

                                     P = Runtime.getRuntime().exec(strParam);

             System.out.println("here " + strParam);

             BufferedReader br = new BufferedReader(new InputStreamReader(P.getInputStream()));

            String line=null;

             while ( (line = br.readLine()) != null){

                        System.out.println("Output" + ">" + line);

                  Var=line;

                        }

                System.out.println(line);

                System.out.println("Output of Var:-"  +Var);

            int wait = P.waitFor();

            System.out.println("exit code: "+wait);

            }

            catch (InterruptedException ie)

            {

              System.out.println("exception caught");

            }

            catch (Throwable t)

                  {

                    t.printStackTrace();

                  }

               return Var;

           }

}


Step 6: Create a string variable Var1.

Step 7:  Drag and drop a Java embedding activity in the BPEL.

Java_Embedding2:

Test pocob = new Test();               

try{         

addAuditTrailEntry("Started From This Place");        

String i = ((oracle.xml.parser.v2.XMLElement)getVariableData("inputVariable","payload","/ns1:Input/ns1:num1")).getFirstChild().getNodeValue();    

int k = Integer.parseInt(i);        

addAuditTrailEntry("k Variable " +k);        

String j = ((oracle.xml.parser.v2.XMLElement)getVariableData("inputVariable","payload","/ns1:Input/ns1:num2")).getFirstChild().getNodeValue();           

int n = Integer.parseInt(j);        

addAuditTrailEntry("n Variable " +n);        

addAuditTrailEntry("Before Variable");        

String greetings = pocob.ShellScript(k,n);       

addAuditTrailEntry("After Variable");        

addAuditTrailEntry("Value:- " +greetings);        

setVariableData("Var1",greetings);     //Var1 is a variable created in BPEL process to contain result     

addAuditTrailEntry("After Assignment");        

}        catch(Exception io){                                                                          

      addAuditTrailEntry("Exception occured:"+io.getMessage());                                                          

      javax.xml.namespace.QName qName=new javax.xml.namespace.QName("http://schemas.oracle.com/bpel/extension","remoteFault");                                                                               

      com.oracle.bpel.client.BPELFault bpelFault=new com.oracle.bpel.client.BPELFault(qName);               

      throw bpelFault;                                 

    }       

finally{        

pocob=null;   

}


Step 8 : Import the java class in the BPEL. location used as packageName.ClassName

<import location="invokeshellscriptproject.Test" importType="http://schemas.oracle.com/bpel/extension/java"/>

Step 9 : Assign Var1 to result variable.

Tuesday, September 8, 2020

Bash scripting and shell programming linux part 3

The Part#3 includes:

  • Variable scope
  • let command
  • case statements
  • eval command
  • wildcards
  • Logger
  • Reading a file line by line
  • expr command
  • loop control statements - Break and Continue
  • Debugging

variable scope:

Global:

  • By default, variables are global.
  • define variables before use it
  • If you define a global variable in a function, it will be only be available after the function is called.
Local:
  • Can only be accessed within a function using a local keyword, example: local LOCAL_VARIABLE=1
let command:
  • The let command is an arithmetic operator. It is almost same as (( )). Only difference is that, let is an arithmetic command while (( )) is a compound command.
  • It is a built-in command which instructs shell to perform an evaluation of arithmetic expressions. No spaces should be used around the arithmetic operand with let command.

 let y=y+1
 echo $y
1
let y="34 +45"
echo $y
79

case statements:
A case construct helps us to simplify nested if statement. You can match several variables against one variable. Each case is an expression matching a certain pattern.

Syntax:
case $var in
pattern_1)
    Commands
;;
pattern_N)
    Commands
;;
esac

Example:
cat > case.sh
#!/usr/bin/bash
echo "Enter the name of the State"
read state
case $state in
"Uttar Pradesh")
echo "Capital id Lucknow"
;;
"Bihar")
echo " Capital id Patna"
;;
*)
echo " You discovered an unknown state"
;;
esac

chmod +x case.sh
./case.sh
Enter the name of the State
Bihar
 Capital id Patna
./case.sh
Enter the name of the State
bihar
 You discovered an unknown state

eval command:
The eval command is a built-in command. It takes a string as its argument and evaluate it, then run the command stored in the argument. It allows using the value of a variable as a variable.
eval echo \${$User}


cat > eval.sh
#!/usr/bin/bash
hello=Mr.x
user=hello
echo \${$user}
eval echo "\${$user}"

chmod +x eval.sh
./eval.sh
$[hello}
Mr.x

But command "eval echo \${$User}" runs the parameter passed to eval. After expansion, remaining parameters are echo and ${Hello}. Hence eval command runs the command echo ${Hello}. And so the output is Mr. X

Wild cards:
  • A character or string used for pattern matching.
  • wildcards can be used with most of the commands like cp, ls, rm etc.
* : Matches zero or more characters.
? : Matches exactly one character

Character classes[]: Matches any of the characters included between the brackets.matches exactly one character.
Example:
[aeiou]
ca[nt]* : can, cat, candy, catter etc.
Character classes [!]: Matches any of the characters not included between the brackets. matches exactly once character.
Example:
[!aeiou] : Devdas, criccket.

Ranges: use two chars separated by a hypen to create a range in a character class.
[a-g]* : matches all files starting with a,b,c,d,e,f,g
[1-5]*: matches all files started with 1,2,3,4,5

Named character class: 
[[:alpha:]]
[[:alnum:]]
[[:digit:]]
[[:lower:]]
[[:upper:]]
[[:space::]]

escape character: "\" is called escape character. use it to match a wildcard character.
Example:
*\? : match all files that end with a question mark.


Example:
 ls -l a*
-rwxr-xr-x    1 oracle   dba              22 Aug 11 03:20 a.sh

ls c[aeiou]*
case.sh    config.sh

ls c[!aeiou]*
c.sh

ls [a-d]*
a.sh       b.sh       c.sh       case.sh    config.sh

ls [[:digit:]]*
20200819mycat.jpg

ls *.txt
test.txt

ls ?est.txt
test.txt

copying all the html files from current directory to /a directory.
#!/usr/bin/bash
for FILE in *.html
do
echo "copying $FILE"
cp $FILE /a
done

Logging:
Syslog:
  • The syslog standards uses facilities and severities to categorize messages.
    • Facilities: kern,mail,user,daemon,auth,local0
    • Severities: alert, emerg,crit,err,warning,notice,info debug
  • Log files locations are configureables
    • /var/log/messages
    • /var/log/syslog
Logging with logger:
logger "Message"
logger -p local0.info "Message"
logger -t myscript -p local0.info "Message"
logger -i -t myscript -p local0.info "Message"

Reading a file line by line:
#!/usr/bin/bash
LINE_NUM=1
while read LINE
do
echo "$LINE_NUM : $LINE"
((LINE_NUM++))
done < /test/test.txt

expr command:

  • The expr command in Unix evaluates a given expression and displays its corresponding output. It is used for: Basic operations like addition, subtraction, multiplication, division, and modulus on integers.
  • Evaluating regular expressions, string operations like substring, length of strings etc.
Syntax:

$expr expression


Performing operations on variables inside a shell script

Example: Adding two numbers in a script

echo "Enter two numbers"
read x 
read y
sum=`expr $x + $y`
echo "Sum = $sum"

Note: expr is an external program used by Bourne shell. It uses expr external program with the help of backtick. The backtick(`) is actually called command substitution.

Comparing two expressions
Example:
x=10
y=20
# matching numbers with '='
res=`expr $x = $y`
echo $res

# displays 1 when arg1 is less than arg2
res=`expr $x \< $y`
echo $res

# display 1 when arg1 is not equal to arg2
res=`expr $x \!= $y`
echo $res

For String operations
Example: Finding length of a string
x=geeks
len=`expr length $x`
echo $len

Example: Finding substring of a string
x=geeks
sub=`expr substr $x 2 3` 
#extract 3 characters starting from index 2
echo $sub

Break Statement:
The break statement is used to terminate the execution of the entire loop, after completing the execution of all of the lines of code up to the break statement. It then steps down to the code following the end of the loop.

Syntax:
The following break statement is used to come out of a loop −
break

The break command can also be used to exit from a nested loop using this format −
break n
Here n specifies the nth enclosing loop to the exit from.

Example 1:
Here is a simple example which shows that loop terminates as soon as a becomes 5 −

#!/usr/bin/bash
a=0
while [ $a -lt 10 ]
do
echo "$a"
if [ $a -eq 5 ]
then 
break
fi
a=`expr $a + 1`
done

Upon execution, you will receive the following result −
0
1
2
3
4
5
Example 2 of nested for loop. This script breaks out of both loops if var1 equals 2 and var2 equals 0 −

#!/usr/bin/sh
for var1 in 1 2 3
do
   for var2 in 0 5
   do
      if [ $var1 -eq 2 -a $var2 -eq 0 ]
      then
         break 2
      else
         echo "$var1 $var2"
      fi
   done
done

Upon execution, you will receive the following result. In the inner loop, you have a break command with the argument 2. This indicates that if a condition is met you should break out of outer loop and ultimately from the inner loop as well.

1 0
1 5

The continue statement:
The continue statement is similar to the break command, except that it causes the current iteration of the loop to exit, rather than the entire loop.

This statement is useful when an error has occurred but you want to try to execute the next iteration of the loop.

Syntax
continue
Like with the break statement, an integer argument can be given to the continue command to skip commands from nested loops.

continue n
Here n specifies the nth enclosing loop to continue from.

Example 1:
The following loop makes use of the continue statement which returns from the continue statement and starts processing the next statement −

#!/bin/sh
NUMS="1 2 3 4 5 6 7"
for NUM in $NUMS
do
   Q=`expr $NUM % 2`
   if [ $Q -eq 0 ]
   then
      echo "Number is an even number!!"
      continue
   fi
   echo "Found odd number"
done

Upon execution, you will receive the following result −
Found odd number
Number is an even number!!
Found odd number
Number is an even number!!
Found odd number
Number is an even number!!
Found odd number

Debugging:
Debugging is required to fix bugs or errors and to determine the root of unexpected behavior.

Technique 1:
cat >debug.sh
#!/usr/bin/bash -x
VAR_TEST="test"
echo "$VAR_TEST"

chmod +x debug.sh
 ./debug.sh

Output:
+ VAR_TEST=test
+ echo test
test

Technique 2:
#!/usr/bin/bash
VAR_TEST="test"
set -x
echo $VAR_TEST
set +x
hostname

output:
+echo test
linuxTestServ

-e option:
  • Exit on error
  • can be used with other options
    • #!/usr/bin/bash -ex
    • #!/usr/bin/bash -xe
    • #!/usr/bin/bash -e -x
    • #!/usr/bin/bash -x -e
Example 1:
#!/usr/bin/bash -e
FILE_NAME="/invalid"
ls $FILE_NAME
echo $FILE_NAME

Output:
ls: 0653-341 The file /invalid does not exist.
/invalid

Example 2:
#!/usr/bin/bash -ex
FILE_NAME="/invalid"
ls $FILE_NAME
echo $FILE_NAME

Output:
+ FILE_NAME=/invalid
+ ls /invalid
ls: 0653-341 The file /invalid does not exist.

-v option: Prints shell input lines as they are read.
#!/usr/bin/bash -v
VAR_TEST="test"
echo "$VAR_TEST"

Output:
#!/usr/bin/bash -v
VAR_TEST="test"
echo "$VAR_TEST"
test

#!/usr/bin/bash -vx
VAR_TEST="test"
echo "$VAR_TEST"

Output:
#!/usr/bin/bash -vx
VAR_TEST="test"
+ VAR_TEST="test"
echo "$VAR_TEST"
+ echo test
test

PS4:
  • Controls what is displayed before a line when using teh "-x" option.
  • the default value is "+"
  • Bash variables BASH_SOURCE, LINENO etc.
Example:
cat > ps4.sh
#!/usr/bin/bash -x
PS4='+ $BASH_SOURCE : $LINENO : '
TEST_VAR="test"
echo "$TEST_VAR"

chmod +x ps4.sh
./ps4.sh
+ PS4='+ $BASH_SOURCE " $LINENO " '
+ ./ps4.sh : 3 : TEST_VAR=test
+ ./ps4.sh : 4 : echo test
test

#!/usr/bin/bash -x
PS4='+ $BASH_SOURCE : $LINENO : $FUNCNAME[0]()'
debug(){
echo "Executing: $@"
$@
}
debug ls


+ PS4='+ $BASH_SOURCE : $LINENO : $FUNCNAME[0]()'
+ ./ps41.sh : 7 : [0]()debug ls
+ ./ps41.sh : 4 : debug[0]()echo 'Executing: ls'
Executing: ls
+ ./ps41.sh : 5 : debug[0]()ls
20200819mycat.jpg 


DOS VS Linux  file types:
A file can contain CRLF/Carriage return, Line feed. So when we are moving file from DOS system to Linux, we should use " dos2unix script.sh " command to make it compatible with BASH executable.

To continue, go to the next page, bash-scripting-and-shell-programming-part4.

Monday, September 7, 2020

Bash scripting and shell programming linux part 2

 The Part#2 includes:

  • Positional Parameters
  • Shell Scripting Shift Through Parameters
  • Shell Scripting Sourcing a file
  • Accepting User input (STDIN)
  • Shell Scripting Sourcing a config file
  • Return codes / Exit statuses
  • EXIT Command
  • Functions

Positional Parameters:

  • A bash shell script have parameters. These parameters start from $1 to $9.
  • When we pass arguments into the command line interface, a positional parameter is assigned to these arguments through the shell.
  • The first argument is assigned as $1, second argument is assigned as $2 and so on...
  • If there are more than 9 arguments, then tenth or onwards arguments can't be assigned as $10 or $11.
  • You have to either process or save the $1 parameter, then with the help of shift command drop parameter 1 and move all other arguments down by one. It will make $10 as $9, $9 as $8 and so on.


$1-$9 Represent positional parameters for arguments one to nine
${10}-${n} Represent positional parameters for arguments after nine
$0 Represent name of the script
$∗ Represent all the arguments as a single string
$@ Same as $∗, but differ when enclosed in (")
$# Represent total number of arguments
$$ PID of the script
$? Represent last return code

Example:
cat > positional.sh
#!/usr/bin/bash
echo "The script name: $0"
echo "The 1st argument: $1"
echo "The 2nd argument: $2"
echo "The pid of the script: $$"
echo "The total numbe rof arguments: $#"
echo "All the arguments: $*"
echo "The last return code: $?"

chmod +x positional.sh
./positional.sh
The script name: ./positional.sh
The 1st argument: 1
The 2nd argument: 4
The pid of the script: 4129020
The total numbe rof arguments: 2
All the arguments: 1 4
The last return code: 0


Shell Scripting Shift Through Parameters:
Shift command is a built-in command. Command takes number as argument. Arguments shift down by this number.
For example, if number is 5, then $5 become $1, $6 become $2 and so on.

The shift command is mostly used when arguments are unknown. Arguments are processed in a while loop with a condition of (( $# )). this condition holds true as long as arguments are not zero. Number of arguments are reduced each time as the shift command executes.
cat > shift.sh
#!/usr/bin/bash
if [ "$#" == "0" ]
then
echo pass at least one parameter
exit 1
fi
while (( $# ))
do
echo you gave me $1
shift
done
 chmod +x shift.sh
 ./shift.sh
pass at least one parameter
./shift.sh 1 e
you gave me 1
you gave me e


Shell Scripting Sourcing a file:
A file is sourced in two ways. 
  • source <fileName>
  •   . ./<filename> 
When a file is sourced, the code lines are executed as if they were printed on the command line.
The difference between sourcing and executing a script is that, while executing a script it runs in a new shell whereas while sourcing a script, file will be read and executed in the same shell.

source ./variable.sh
jai viru
echo $var1
jai
. ./variable.sh
jai viru
echo $var2
viru

Accepting User input (STDIN):
The read command allows a user to provide the runtime input.

 cat > read.sh
#!/usr/bin/bash
echo enter your name:
read name
echo "Welcome $name "
chmod +x read.sh
 ./read.sh
enter your name:
sri
Welcome sri

Syntax:
read -p "Prompt" variable

Example:
#!/usr/bin/bash
read -p "Enter your name" Name
echo " Hello $Name"

Output:
Enter your name
rabi
Hello rabi


Shell Scripting Sourcing a config file:
Many programs use external configuration files. Use of external configuration files prevents a user from making changes to a script. Config file is added with the help of source command.
If a script is shared in many users and every user need a different configuration file, then instead of changing the script each time simply include the config files.

We have two files, one is parent file (main.sh) and other is configuration file (config.sh). We have to source this configuration file into our parent file
Example 1:

Config.sh
#!/usr/bin/bash
user =java
id=201

Main.sh
#!/usr/bin/bash
source config.sh
echo “this is user $user with $id”
chmod +x main.sh
 ./main.sh
 this is user abc with 101.
Note: We can also use ( . config.sh ) command instead of ( source config.sh ) .

Example 2:

cat > config.sh
#! /usr/bin/bash
user=abc
id=101

 cat > main1.sh
#! /usr/bin/bash
. config.sh
echo "this is user $user"
chmod +x main1.sh
./main1.sh
Output:
this is user abc

Return codes / Exit statuses:
  • Every command returns an exit codes, range from 0 to 255.
  • 0=success and other than zero is a error condition.
  • We can use this exit status as error checking purposes.
  • $? returns the return code of the previously executed command.
Example:
#!/usr/bin/bash
HOST="google.com"
ping -c 1 $HOST
RETURN_CODE=$?
if [ $RETURN_CODE -eq "0" ]
then 
echo "$HOST is reachable"
else
echo "$HOST is not reachable"
fi
chmod +x ret.sh
./ret.sh
PING google.com: (xxx.xxx.x.xx): 56 data bytes
64 bytes from xxx.xxx.x.xx: icmp_seq=0 ttl=111 time=42 ms

--- google.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 42/42/42 ms
google.com is reachable

&& and ||:
#!/usr/bin/bash
HOST="google.com"
#If 1st condition is true then it executes the second command.
ping -c 1 $HOST && echo " $HOST is reachable"
#If 1st condition is false then it executes the second command.
ping -c 1 $HOST || echo "$HOST is unreachbale"

The Semicolon:
It separates the commands with a semicolon to ensure they all get executed.
cp test.txt /test/bak/ ; cp tets.txt  /test/

EXIT Command:
Using EXIT keyword we can explicitly define the return code. The default value is that of the last command executed. exit 0, exit 1, ....exit 255 etc.
Example:
#!/usr/bin/bash
HOST="google.com"
ping -c 1 $HOST
if [ $HOST -ne "0" ]
then 
echo " $HOST unreachable"
exit 1
fi
exit 0

More examples:
 ping -c 1 abc.com
PING abc.com: : 56 data bytes
64 bytes from : icmp_seq=0 ttl=238 time=42 ms

--- abc.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 42/42/42 ms
 echo $?
0
ping -c 1 ttrert.com
0821-062 ping: host name ttrert.com NOT FOUND
echo $?
1
 ping -c 1 amazon.com.1
0821-062 ping: host name amazon.com.1 NOT FOUND
 echo $?
127

Functions:
With the help of functions, overall functionality of a function can be divided into smaller or logical parts, which can be called to perform their task. It helps us to check our program part by part. We can reuse the function where ever we want.
  • Don't Repeat yourself(DRY)
  • Write once, use many times
  • easier to maintain and troubleshoot
Creating a function:
Syntax:
function functionName () {  
        Commands to be executed  
}
or
functionName () {  
        Commands to be executed  
}
You will call your function with their function name.

Example 1:
cat > function.sh
#!/usr/bin/bash
function helloWorld(){
echo "hello"
}
helloWorld

chmod +x function.sh
./function.sh
hello

Example 2:
cat > functest.sh
#!/usr/bin/bash
function hello(){
for NAME in $@
do
echo "Hello $NAME"
done
}
hello sri jinx pepa luis

chmod +x functest.sh
./functest.sh
Hello sri
Hello jinx
Hello pepa
Hello luis


Passing Parameters:
You can pass one or more parameters in a function. Parameters will be defined as $1, $2 and so on.

 cat > funcparam.sh
#!/usr/bin/bash
function passParam(){
echo "param1= $1 and param2= $2"
}
#call function
passParam test 123

 chmod +x funcparam.sh
 ./funcparam.sh
param1= test and param2= 123

Returning Values from Functions:
  • If you execute an exit command from inside a function, its effect is not only to terminate execution of the function but also of the shell program that called the function.
Based on the situation you can return any value from your function using the return command whose syntax is as follows −
return code

cat > funcreturn.sh
#!/usr/bin/bash
Hello(){
echo "hello $1 $2"
return 10
}
Hello Sri Das
ret=$?
echo "return vaue is $ret"

chmod +x funcreturn.sh
./funcreturn.sh
hello Sri Das
return vaue is 10

Nested Functions:
One of the more interesting features of functions is that they can call themselves and also other functions. A function that calls itself is known as a recursive function.
Following example demonstrates nesting of two functions −

#!/bin/sh
# Calling one function from another
number_one () {
   echo "This is the first function speaking..."
   number_two
}

number_two () {
   echo "This is now the second function speaking..."
}

# Calling function one.
number_one

To continue, go to the blog page  part 3: bash-scripting-and-shell-programming_part3

Wednesday, September 2, 2020

Bash scripting and shell programming linux part 1

The Part#1 includes:

  • What is a Shell & Types of shells
  • What is Script and how to determine shell
  • She-Bang & comments
  • Steps to create a shell script and execution
  • Variables & usages, assign output command to a variable, Variables Names
  • File and String Test Operators
  • Decision making statements if, if else, if elif else
  • Shell Scripting loops for, while, infinite loop, until

 What is a Shell?

An Operating system is made of many components, but its two prime components are -

  • Kernel
  • Shell

A Kernel is at the nucleus of a computer. It makes the communication between the hardware and software possible. While the Kernel is the innermost part of an operating system, a shell is the outermost one.

A shell in a Linux operating system takes input from you in the form of commands, processes it, and then gives an output. It is the interface through which a user works on the programs, commands, and scripts. A shell is accessed by a terminal which runs it. When you run the terminal, the Shell issues a command prompt (usually $), where you can type your input, which is then executed when you hit the Enter key. The output or the result is thereafter displayed on the terminal.

The Shell wraps around the delicate interior of an Operating system protecting it from accidental damage. Hence the name Shell.

A shell is a command-line interpreter and typical operations performed by shell scripts include file manipulation, program execution, and printing text.

Types of Shell

There are two main shells in Linux:

1. The Bourne Shell: The prompt for this shell is $ and its derivatives are listed below:

  • POSIX shell also is known as sh
  • Korn Shell also knew as sh
  • Bourne Again SHell also knew as bash (most popular)

2. The C shell: The prompt for this shell is %, and its subcategories are:

  • C shell also is known as csh
  • Tops C shell also is known as tcsh

Scripts:

  • contains a series of commands
  • An interpreter executes commands in the scripts
  • anything you can type at the command line, you can put in a script.
  • great for automating tasks.
How to determine Shell:
oracle@test> echo $SHELL
/usr/bin/bash

The $ sign stands for a shell variable, echo will return the text whatever you typed in.

Shell Scripting She-bang:
The sign #! is called she-bang and is written at top of the script. It passes instruction to program /bin/sh.
To run your script in a certain shell (shell should be supported by your system), start your script with #! followed by the shell name.
Example:
#!/bin/bash  
echo Hello World  
  • The first line of a shell script typically starts with a shebang followed by the path to an interpreter that will be used to execute the commands in the script. 
  • If you do not supply a shebang and specify an interpreter on the first line of the script the commands in the script will be executed using your current shell.

Shell Scripting Comments:
Any line starting with a hash (#) becomes comment. Comment means, that line will not take part in script execution. It will not show up in the output.

Steps to create a script and execute:
  • create a .sh file using cat or touch command
  • write the body and saved it
  • provide the execute permission using chmod 777 or chomd +x command
  • execute by ./scriptname.sh
Example:
cat > variable.sh

#!/usr/bin/bash

var1=jai

var2=viru

#Echo

echo "$var1 $var2"

chmod 777 variable.sh

./variable.sh

Output: jai viru

Variables:

  • Storage location that have a name. By convention variables are upper case.
  • Name value pair like variable_name = "Value"
  • Variables are case sensitive

Variable usage:

#! /bin/bash

MY_SHELL="bash"

echo " I like the $MY_SHELL shell"

or 

echo "I like the ${MY_SHELL} shell"

Output:

I like the bash shell

Assign command output to a variable:

#!/bin/bash

SERVER=$(hostname)

echo "The script is running on ${SERVER}"

or 

#!/bin/bash

SERVER=`hostname`

echo "The script is running on ${SERVER}"

Output:

The script is running on LinxTestServer

Variables Names:

  • It contains combination of uppercase and lowercase alphabets, digits and underscore as follows:

FIRST3VARIABLE

FIRST_VARIABLE

FirstVarable

  • Variable names should not start with a digit or not use any special characters other than underscore. For instance,

2FirstVariable

First-Varable

FIRST@VARIABLE

Read-only Variables:

Shell provides a way to mark variables as read-only by using the read-only command. After a variable is marked read-only, its value cannot be changed.

For example, the following script generates an error while trying to change the value of NAME −

 cat > read.sh

#!/usr/bin/bash

NAME="Sri"

readonly NAME

NAME="Rudra"

echo "$NAME"

The above script will generate the following result −

./read.sh: line 4: NAME: readonly variable

Sri

Unsetting Variable:

Unsetting or deleting a variable directs the shell to remove the variable from the list of variables that it tracks. Once you unset a variable, you cannot access the stored value in the variable.

Following is the syntax to unset a defined variable using the unset command −

unset variable_name

The above command unsets the value of a defined variable. Here is a simple example that demonstrates how the command works −

 cat > read1.sh

#!/usr/bin/bash

NAME="Sri"

unset NAME

echo $NAME

The above example does not print anything. You cannot use the unset command to unset variables that are marked readonly. 

 File Operators(Tests):

-d FILE : True if file is a directory

-e FILE : True if file exists

-f FILE : True if file exists and is a regular file

-r FILE : True if the file is readable by you

-s FILE : True if file exists and not empty

-w FILE : True if file is writable by you

-x FILE : True if is executable by you 

String Operators(tests):

-z STRING : True if string is empty

-n STRING : True if string is not empty

String1 = String2 : True if the strings are equal

String1 != String2 : True if the strings are not equal.

Making Decisions:

The if statement:

syntax:

if [ condition is true ]

then 

commands

fi

Example:

#! /bin/bash

MY_SHELL="bash"

if [ $MY_SHELL = "bash" ]

then 

echo "You like the bash shell"

fi


if-else statement:

Syntax:

if [ condition is true ]

then

commands

else

commands

fi

Example:

#! /bin/bash

MY_SHELL="csh"

if [ $MY_SHELL="bash" ]

then

echo " You like bash shell"

else

echo " you dont like bash shell"

fi

if elif else statements:

Syntax:

if [ condition is true ]

then

commands

elif [ condition is true ]

then 

commands

else

commands

fi

Example:

#! /bin/bash

MY_SHELL="bash"

if [ $MY_SHELL="bash" ]

then

echo " You like bash shell"

elif [ $MY_SHELL="csh" ]

then

echo " you like csh shell"

else

echo " you dont like bash or csh shells"

fi

Shell Scripting for loop:

The for loop moves through a specified list of values until the list is exhausted.

Syntax 1:

Syntax of for loop using in and list of values is shown below. This for loop contains a number of variables in the list and will execute for each item in the list. For example, if there are 10 variables in the list, then loop will execute ten times and value will be stored in varname.

for table in {min_value..Max_value..Increment_value}

do

commands

done

Example:

cat >for1.sh

#!/usr/bin/bash

for table in {2..20..2}

do

echo "$table"

done

chmod +x for1.sh

 ./for1.sh

Output:

2

4

6

8

10

12

14

16

18

20

Syntax 2:

for (( initialization;condition;increment/decrement))

do

commands

done

Example:

cat > for2.sh

#!/usr/bin/bash

for (( i=10; i >= 1; i--))

do

echo "$i"

done

chmod +x for2.sh

./for2.sh

Output:

10

9

8

7

6

5

4

3

2

1

Syntax 3:

for Var_name in Item_1 ItemN

do

commands

done

Example:

#! /bin/bash

COLORS="RED GREEN BLUE"

for COLOR in $COLORS

do 

echo "COLOR: $COLOR"

done

Output:

COLOR:RED
COLOR:GREEN
COLOR:BLUE


Scripting while loop:

There is a condition in while. And commands are executed till the condition is valid. Once condition becomes false, loop terminates.

Syntax:

While [ condition ]

Do

commands

done

example:

cat >while.sh

#!/usr/bin/bash

i=10;

while [ $i -ge 0 ] ;

do

echo " reverse order number $i"

let i--;

done

chmod +x while.sh

 ./while.sh

Output: 

reverse order number 10

 reverse order number 9

 reverse order number 8

 reverse order number 7

 reverse order number 6

 reverse order number 5

 reverse order number 4

 reverse order number 3

 reverse order number 2

 reverse order number 1

 reverse order number 0

while infinite loop:

Infinite loop is also called endless loop. It is made with while true (it means condition will always be true) or while : (it means an empty expression), where colon (:) is equivalent to no operation.

 cat whileinfinite.sh

#!/usr/bin/bash

while true

do

echo " this is infinite loop. press ctrl + c to exit"

done

Shell Scripting until loop:

It is similar to while loop. The only difference is that until statement executes its code block while its conditional expression is false, and while statement executes its code block while its conditional expression is true.

Difference between while and until:

Until loop always executes at least once. Loop while executes till it returns a zero value and until loop executes till it returns non-zero value.

Syntax:

Until [ condition ]

Do

Commands

Done

Example:

 cat > until.sh

#!/usr/bin/bash

i=5;

until [ $i -gt 15 ];

do

echo "number $i"

let i++;

done

 chmod +x until.sh

 ./until.sh

Output:

number 5

number 6

number 7

number 8

number 9

number 10

number 11

number 12

number 13

number 14

number 15


Example: Rename all the.jpg files of the current folder to YYYY-MM-DD- filename.
#! /bin/bash
LIST_PICTURES=$(ls *.jpg)
DATE=`date +%F`
for PICTURE in LIST_PICTURES
do
echo "Renaming $PICTURE to $DATE-$PICTURE"
mv $PICTURE $DATE-$PICTURE
done

To continue, go to the next blog page bash-scripting-and-shell-programming-part-2.


Featured Post

11g to 12c OSB projects migration points

1. Export 11g OSB code and import in 12c Jdeveloper. Steps to import OSB project in Jdeveloper:   File⇾Import⇾Service Bus Resources⇾ Se...