In my previous two posts (1 and 2 below) we looked at vi basic and editing.
1. Essentials of Vi editor - part 1
2. Essentials of Vi editor - part 2
In this post let's look at few advanced commands and techniques we can use in vi editor
Indentation and work wrapping
-----------------------------------------------------
>> indent the current line. e.g 3>> indents 3 lines, >} indents a paragraph
<< outdents the current line
:se ai // Enables auto indent
:se noai // Disables auto indentation
:se wm=8 // Sets the wrap margin to 8. As soon as you enter 8 columns it wraps the cursor
:se wm=0 // Auto wrap set off
Filtering
-----------------------------------------------------
!! - applies the filter to the current line
n!! - applies the filter to n number of lines from current line
!} - filters the next paragraph
!{ - filter the previous paragraph
!% - applies the filter from current location to next parranthesis, brace or bracket
These filters can be can be applied to shell commands like tr (transformation), fmt (formatting), grep (search), sed (advanced editing) or awk (a complete programming language). This would mean like sending the filtered text through these commands and getting the output of it and searching or placing back in file as applicable.
e.g In command mode uf you type
!!tr a-z A-z // And enter. Turns the current line into uppercase. Note however your lower command shows :.!tr a-z A-Z. It converts into a format that vi understands it.
5
Advanced examples with : command
-----------------------------------------------------
: 1,. d // Delete all the lines from the first line (indicated by 1) to current line (indicated by .)
: .,$ s/test/text/g // From current line (indicated by .) to end of line (indicated by $) search and replace all 'test' occurrences to 'text'
: /first/,/last/ ! tr a-z A-Z // Find first line that matches 'first' regexp to the first match following 'last' regex and filter it (indicated by !) using the unix command tr from a-z to A-Z (means convert to upper case)
ma // marks a line by character 'a'
mb // marks a line by character 'b'
!a // jump to the line marked by a
: 'a,'b d // Delete all the lines marked by a through b
------- Update - Youtube tutorial of the above (04/01/2023) ---------
Youtube Video
Sunday, December 25, 2016
Essentials of Vi editor - part 2
In the previous post we looked at some of the basics of the vi editor. In this post let's walk through searching, replacing and undoing.
Search and Replace
------------------------------------------------------------------
/text - searches the text.
?text - searches backward
n - repeats the previous search
N - repeats the previous search in backward direction
. matches any single character e.g - /a.c matches both abc, adc etc. Doesn't match 'ac'
\ has special meaning. e.g - /a\.c matches a.c exactly
e.g - /a\\c matches a\c, /a\/c matches a/c
^ - matches line beginning. e.g - /^abc matches lines beginning with abc
$ - matches line ending e.g - /xyz$ matches lines ending with xyz
[] - matches single character in a set. e.g - /x[abc]y matches xay, xby and xcy
e.g - /x[a-z]y matches xzy, xay etc
e.g - /x[a-zA-Z]y matches xay, xAy etc
e.g - /x[^a-z]y // matches x followed by anything other than a lowercase letter followed by y. Therefore 'xay' doesn't match, but xBy matches.
* - zero or more matches. e.g - xy*z matches xyz, xyyyyz and also xz
\( \) - e.g - /\(xy\)*z matches xyz, xyxyx, xyxyxyz etc
/<.*> - matches <iwhewoip> and <my_name> etc
/<[^>]*> - matches anything in between <>
:s/old/new/ - replaces the first occurrences of old to new on current line
:s/old/new/g - replaces all in the current line
:%s/old/new/ - replaces the first occurrences of old to new of every line in the document
:%s/old/new/g - globally replace all occurrences in the document
You may use any special character other than / for delimitation. For example you may use | or ;
Few special examples.
:%s/test/(&)/g - Here the replacement string is (&), Here the & says the current match. Therefore whatever test words in the document will be put into parenthesis as in (test)
Undoing
------------------------------------------------------------------
u - undoing the last change in command mode
Ctrl + r - Redo the last change
U - undo all fhe changes in the current line
. (period) - Repeats last change in your cursor locations
yy - yanks (copy) a line (Similar to dd like delete equivalents) - the yanked texts goes to vi's buffer not to the OS clip-board
yw - yanks a word (just like dw deletes a word)
p - pastes the yanked text after the cursor
P - pastes the yanked text before the cursor
------- Update - Youtube tutorial of the above (04/01/2023) ---------
Youtube Video
Search and Replace
------------------------------------------------------------------
/text - searches the text.
?text - searches backward
n - repeats the previous search
N - repeats the previous search in backward direction
. matches any single character e.g - /a.c matches both abc, adc etc. Doesn't match 'ac'
\ has special meaning. e.g - /a\.c matches a.c exactly
e.g - /a\\c matches a\c, /a\/c matches a/c
^ - matches line beginning. e.g - /^abc matches lines beginning with abc
$ - matches line ending e.g - /xyz$ matches lines ending with xyz
[] - matches single character in a set. e.g - /x[abc]y matches xay, xby and xcy
e.g - /x[a-z]y matches xzy, xay etc
e.g - /x[a-zA-Z]y matches xay, xAy etc
e.g - /x[^a-z]y // matches x followed by anything other than a lowercase letter followed by y. Therefore 'xay' doesn't match, but xBy matches.
* - zero or more matches. e.g - xy*z matches xyz, xyyyyz and also xz
\( \) - e.g - /\(xy\)*z matches xyz, xyxyx, xyxyxyz etc
/<.*> - matches <iwhewoip> and <my_name> etc
/<[^>]*> - matches anything in between <>
:s/old/new/ - replaces the first occurrences of old to new on current line
:s/old/new/g - replaces all in the current line
:%s/old/new/ - replaces the first occurrences of old to new of every line in the document
:%s/old/new/g - globally replace all occurrences in the document
You may use any special character other than / for delimitation. For example you may use | or ;
Few special examples.
:%s/test/(&)/g - Here the replacement string is (&), Here the & says the current match. Therefore whatever test words in the document will be put into parenthesis as in (test)
Undoing
------------------------------------------------------------------
u - undoing the last change in command mode
Ctrl + r - Redo the last change
U - undo all fhe changes in the current line
. (period) - Repeats last change in your cursor locations
yy - yanks (copy) a line (Similar to dd like delete equivalents) - the yanked texts goes to vi's buffer not to the OS clip-board
yw - yanks a word (just like dw deletes a word)
p - pastes the yanked text after the cursor
P - pastes the yanked text before the cursor
------- Update - Youtube tutorial of the above (04/01/2023) ---------
Youtube Video
Essentials of Vi editor - part 1
I'm sure if you are a serious programmer that you would agree vi is a programmers editor. Knowing vi's commands and usage helps you a lot with your programming tasks and undoubtedly it's a light weight powerful toolkit in your arsenal. In this post I would like to refresh your know-how on vi commands and usage, although there are hundreds of posts and cheat sheets available online. Some commands are extremely common whereas there are few I think which is not so common but extremely powerful. I cover these using three posts.
Moving around the files
--------------------------------------------------------------
H (left) , J (down), K (up) , L (right)
w (move forward by one word), b (move backward by one word)
e (move forward till end of current word)
) (Forward a sentence), ( (Bacward a sentence)
} (Forward a full paragraph), { (Backward a full paragraph)
^ (Move to beginning of a line)
$ (Move to end of a line)
% (Matching bracket or brace)
Shift+g (Jump to end of file)
1 and then Shift+g (Jump to beginning of the file)
This works to jump on to a line as well.
e.g: 23 and then Shift+g (Jump to line 23)
Ctrl+e // scroll down one line
Ctrl+y // Scroll up one line
Ctrl+d // Scroll down half a screen
Ctrl+u // Scroll up half a screen
Ctrl+f // Scroll down a full screen
Ctrl+b // Scroll up a full screen
Getting the status of the file
--------------------------------------------------------------
Ctrl+g // Shows if the file is modified, the number of lines and the percentage the current line is from the beginning)
Saving and quitting
--------------------------------------------------------------
:w (Saves the file into the disk and keeps it open)
:wq (Save and close the file) // Equivalent to Shift + ZZ
:q (Quit without saving an unedited file. Warns you if edited)
:q! (Quite without saving even if the file is edited)
:vi <filename> // Closes the current file and open the <filename>. Equivalent to :q and then issuing vi <filename>. If the current file is edited as :q does, a warning will be given
:vi! <filename> // Does the same as above but doesn't warn you. Equivalent to :q! and then issuing vi <filename>.
e.g:
vi file1.txt file2.txt // loads both file into memory and shows the file1.txt
:n // Shift to the next file
:N // Shift back to the previous file
:rew // Rewind to first file if you have multiple files open
:r // read a file and insert into the current file you are editing
Ctrl + g // shows line number and file status
Text Editing
--------------------------------------------------------------
a - append at the end of the cursor
A - append at the end of the line
i - insert before the cursor
I - insert at the beginning of the line
o - Open a new line below
O - open a new line above the current line
All above commands switch the file to insert mode
r - change the current character and be in command mode
s - change the current character and switch to insert mode
cc - delete the current line and switch to insert mode for editing
cw - Edit the current word
c$ - Delete from current position to end of line and keep editing
c^ - Delete all from beginning of the line to current location and keep editing
x - deletes the current character e.g - 5x to delete 5 character
dd - deletes the current line - e.g - 5dd to delete 5 lines
dw - deletes the current word
de - deletes till the end of current word including a whitespace
d^ - deletes from beginning of the line to current caret position
d$ - deletes from current caret position to end of the line (Shift + d does the same)
R - enters to overwrite mode. Whatever character you type will replace the character under the caret
~ - Changes the case of the character under the caret
J - join the next line to the current line
------- Update - Youtube tutorial of the above (04/01/2023) ---------
Youtube Video
Moving around the files
--------------------------------------------------------------
H (left) , J (down), K (up) , L (right)
w (move forward by one word), b (move backward by one word)
e (move forward till end of current word)
) (Forward a sentence), ( (Bacward a sentence)
} (Forward a full paragraph), { (Backward a full paragraph)
^ (Move to beginning of a line)
$ (Move to end of a line)
% (Matching bracket or brace)
Shift+g (Jump to end of file)
1 and then Shift+g (Jump to beginning of the file)
This works to jump on to a line as well.
e.g: 23 and then Shift+g (Jump to line 23)
Ctrl+e // scroll down one line
Ctrl+y // Scroll up one line
Ctrl+d // Scroll down half a screen
Ctrl+u // Scroll up half a screen
Ctrl+f // Scroll down a full screen
Ctrl+b // Scroll up a full screen
Getting the status of the file
--------------------------------------------------------------
Ctrl+g // Shows if the file is modified, the number of lines and the percentage the current line is from the beginning)
Saving and quitting
--------------------------------------------------------------
:w (Saves the file into the disk and keeps it open)
:wq (Save and close the file) // Equivalent to Shift + ZZ
:q (Quit without saving an unedited file. Warns you if edited)
:q! (Quite without saving even if the file is edited)
:vi <filename> // Closes the current file and open the <filename>. Equivalent to :q and then issuing vi <filename>. If the current file is edited as :q does, a warning will be given
:vi! <filename> // Does the same as above but doesn't warn you. Equivalent to :q! and then issuing vi <filename>.
e.g:
vi file1.txt file2.txt // loads both file into memory and shows the file1.txt
:n // Shift to the next file
:N // Shift back to the previous file
:rew // Rewind to first file if you have multiple files open
:r // read a file and insert into the current file you are editing
Ctrl + g // shows line number and file status
Text Editing
--------------------------------------------------------------
a - append at the end of the cursor
A - append at the end of the line
i - insert before the cursor
I - insert at the beginning of the line
o - Open a new line below
O - open a new line above the current line
All above commands switch the file to insert mode
r - change the current character and be in command mode
s - change the current character and switch to insert mode
cc - delete the current line and switch to insert mode for editing
cw - Edit the current word
c$ - Delete from current position to end of line and keep editing
c^ - Delete all from beginning of the line to current location and keep editing
x - deletes the current character e.g - 5x to delete 5 character
dd - deletes the current line - e.g - 5dd to delete 5 lines
dw - deletes the current word
de - deletes till the end of current word including a whitespace
d^ - deletes from beginning of the line to current caret position
d$ - deletes from current caret position to end of the line (Shift + d does the same)
R - enters to overwrite mode. Whatever character you type will replace the character under the caret
~ - Changes the case of the character under the caret
J - join the next line to the current line
------- Update - Youtube tutorial of the above (04/01/2023) ---------
Youtube Video
Thursday, August 18, 2016
Unleashing the Git - part 7 - Cherry picking commits
We looked at git branching in one of my previous post. In this post let's look at git's on of the unique and powerful features that ties with branching; cherry-picking.
You might need to merge only a specific commit in a branch to your local branch, rather than merging an entire branch. Opt to cherry pick only if merging is not an option. One use case where you want to cherry pick is to back port a bug fix you have done in another branch.
Since we covered some branching concepts in the previous post, let's remind a few commands first.
To see all the available branches
You might need to merge only a specific commit in a branch to your local branch, rather than merging an entire branch. Opt to cherry pick only if merging is not an option. One use case where you want to cherry pick is to back port a bug fix you have done in another branch.
Since we covered some branching concepts in the previous post, let's remind a few commands first.
To see all the available branches
$ git branchTo switch to another branch
$ git checkout <branch name>Cherry picking a branch, tag or a specific commit could be done using the following command.
$ git cherry-pick <branch name, tag name or commit id> // In case of branch or tag name, only the latest commit will be picked.Cherry picking also does the commit. You can avoid the commit with the following. This is important when you want to cherry pick multiple commits and commit all stages at once to commit in the current branch.
$ git cherry-pick --no-commit release_branch_v1 // Or same as $ git cherry-pick --n release_branch_v1 $ git cherry-pick --edit <commit id> // launches the editor to change te commit message. Same as $ git cherry-pick -e <commit id> $ git cherry-pick --signoff release_branch_v1 // Adds a “Signed-off-by” line followed by your name and email taken from configuration to the commit message. Same as $ git cherry-pick -s release_branch_v1
Saturday, August 13, 2016
Multimedia in HTML5
Playing audio and video in a web page has not been very straight forward in the past. One reason for that is due to the fact that there are various audio and video containers and encoding formats. Some popular video formats are .avi, .mp4, .ogg, .flv, and .mkv. There is also a new format called WebM, which is likely to get popular. These formats are not encoding formats. These are containers of the encoded content.
When a person making an audio or a video, chooses a format to encode their multimedia content. To view this encoded content, the viewer should have a corresponding decoder installed in their system. This encoder and decoder software is often referred to as a codec (meaning; coder/decoder or compressor/decompressor). Some video encoding formats are H.264, VP8, DivX, Theora, etc. Some audio codecs are MPEG-4 Audio, Layer 3, which you might recognize as MP3, and AAC, which is mostly used by Apple. Vorbis is another audio format, mosly used with an Ogg container. Earlier mentioned WebM is meant to be used exclusively with the VP8 video codec and the Vorbis audio codec.
As mentioned earlier, to view audio/video files, browsers should either support those formats or use a plugin program to decipher these formats. Luckily all popular browsers support or have some plugin program for it (mostly 3rd party programs - e.g: Adobe flash). With HTML5 however, the need for such 3rd party plugin is expected to diminish.
The most common way of including multimedia content into a web page has been to embed an audio/video file and making the user click a button and make the content play within the page. For this to happen, the browser that you are using should support the format of the media. Otherwise you'll have to install a plugin (helper program) to support those formats.
In older browsers the way to include multimedia has been to use the <object> or <embed> tag.
In HTML5, there are new tags for these. <audio> and <video>
So how to cope up with all these formats in new HTML5. It's indeed complex. Answer is to use multiple formats and let your web page in clients browser try to use all those formats, so that there's high chance of making your web page viewer view or listen to the content. So to have multiple formats of your audio/video content, you'll have to use some Software that can convert your main data file to those needed formats. Popular VLC media player can be used for basic usages. You can still include flash formats since flash plugin support is still widely used.
To include a video content, in HTML5 we may use the following format for example.
When a person making an audio or a video, chooses a format to encode their multimedia content. To view this encoded content, the viewer should have a corresponding decoder installed in their system. This encoder and decoder software is often referred to as a codec (meaning; coder/decoder or compressor/decompressor). Some video encoding formats are H.264, VP8, DivX, Theora, etc. Some audio codecs are MPEG-4 Audio, Layer 3, which you might recognize as MP3, and AAC, which is mostly used by Apple. Vorbis is another audio format, mosly used with an Ogg container. Earlier mentioned WebM is meant to be used exclusively with the VP8 video codec and the Vorbis audio codec.
As mentioned earlier, to view audio/video files, browsers should either support those formats or use a plugin program to decipher these formats. Luckily all popular browsers support or have some plugin program for it (mostly 3rd party programs - e.g: Adobe flash). With HTML5 however, the need for such 3rd party plugin is expected to diminish.
The most common way of including multimedia content into a web page has been to embed an audio/video file and making the user click a button and make the content play within the page. For this to happen, the browser that you are using should support the format of the media. Otherwise you'll have to install a plugin (helper program) to support those formats.
In older browsers the way to include multimedia has been to use the <object> or <embed> tag.
In HTML5, there are new tags for these. <audio> and <video>
So how to cope up with all these formats in new HTML5. It's indeed complex. Answer is to use multiple formats and let your web page in clients browser try to use all those formats, so that there's high chance of making your web page viewer view or listen to the content. So to have multiple formats of your audio/video content, you'll have to use some Software that can convert your main data file to those needed formats. Popular VLC media player can be used for basic usages. You can still include flash formats since flash plugin support is still widely used.
To include a video content, in HTML5 we may use the following format for example.
<video src="myvideo.mp4" width="360" height="240" controls></video>Some possible attributes of <video> are -
- autoplay - Plays the video when the page loads. generally not a good idea in usability perspective
- preload - This attribute will pre-load the video. This is good if the video is the central part of your page. Not good if it's not since it'll be a wastage of the bandwidth
- controls - Controls attribute determine if a default set of control buttons should appear. Highly recommended to include this
- loop - Restarts playing the video once it finishes playing
- width - Sets the width of the media box
- height - Sets the height of the media box
<video width="360" height="240" controls> <source src="yourvideo.mp4" type="video/mp4"> <!-- Helps in IE and Safari --> <source src="yourvideo.ogg" type="video/ogg"> <!-- Helps in Chrome and Firefox --> <source src="yourvideo.webm" type="video/webm"> </video>What if someone visits your site with browser that still doesn't support HTML5, we may use the older <embed> tag and use the flash application/x-shockwave-flash type to work with the flash plugin.
<video width="320" height="240" controls> <source src="yourvideo.mp4" type="video/mp4"> <source src="yourvideo.ogg" type="video/ogg"> <source src="yourvideo.webm" type="video/webm"> <embed src="yourvideo.mp4" type="application/x-shockwave-flash" width="320" height="240" allowscriptaccess="always" allowfullscreen="true"> </video>Similarly <audio> tag supports all the parameters as the <video> and the syntax is very similar.
Saturday, July 30, 2016
Securing WSO2 ESB proxies with Kerberos - part 2 (Active directory (AD) as the KDC)
In a previous post, we looked at how to secure a WSO2 ESB proxy services using kerberos in which Key Distribution Center (KDC) is the WSO2 identity server. In this post we'll look at securing WSO2 ESB proxy services when the KDC is the Active Directory (AD).
When the Key Distribution Center (KDC) is the Active Directory (AD), the SPN is usually attached with a service account. Contact your Windows Administrators to create an SPN and a service account and generate a key tab file against that. The main difference from WSO2 IS and AD based KDC is that, in AD the SPN doesn’t provide a text password, rather it’s in the keyTab file.
To create an SPN for a related service account, use the following command.
Ask the administrators to create the keytab using the following command.
Get the keytab file and place it in a location accessible by the WSO2 ESB.
In our case the KDC hostname is us.your.domain.com. The ESB configuration is same as like the previous case. But make the following changes in the krb5.conf
Next when you secure the proxy, you have to provide the SPN name as the AD configured SPN. In our case the SPN is HTTP/USESB@US.YOUR.DOMAIN.COM
Also note since you have no password for the SPN, you can provide the service account password which was created at the AD end. In fact that won’t be used in this case, since the password will get resolved from the keyTab file. Once the proxy is secured, we need to do one more change for Java 7. Go to the proxy dashboard and click Policies. In both, Binding echoSoap11Binding and Binding echoSoap12Binding policies add the following line below the service principal name.
This is required to be added due to a bug that was identified in the implementation and require a patch from WSO2. If you are a support customer of WSO2, you may open a support ticket to request the patch WSO2-CARBON-PATCH-4.2.0-1811.
If you are not a support customer, you can run the ESB using JRE 1.6.
That’s it in ESB end.
When you start the server, start the ESB with the following system property,
./wso2server.sh -Djava.security.krb5.conf= /carbon/wso2/esb/wso2esb-4.8.1/repository/resources/security/krb5.conf start
You may insert that property in the wso2server.sh file as well.
Now you can try invoking the echo proxy using soap ui and it should fail saying “security header does not found”. This is because the request didn’t contain any security information, such as the kerberos ticket etc. In fact it’s difficult to create a soap ui project to work with kerberos, as the WS-Security spec demands a format for the soap request, such as signing the message payload from kerberos ticket and so on.
We have written a java client to cater this. Clients need to incorporate same mechanism to call a kerberos secured proxy. Please find the Java client here.
Note: however the Java client should be running in a machine where it can connect to the AD based KDC.
Testing with a Java client
Change the krb5.conf of the java client as the following in the java client.
The rampart configuration of policy should be changed. Client principal name in this case would be the logged in user. In our case we have the following configurations.
Note the Java7.TicketDecoder property, if you are running the ESB in Java 7 (Note: To use with Java 7, you'll need to have above mentioned patch applied in the ESB 4.8.1 and ESB 4.9.0) If you are running in Java 6, you may remove that line.
If your server is using a custom keystore (which is normally the case), following needs to be done.
1. Use the same keystore, its alias and password of primary keystore for RegistryKeyStore. This is due to a known issue in the system (see [1]). This will get fixed in the future releases.
[1] https://wso2.org/jira/browse/CARBON-15786
2. Change the config.property of the Java client to use the new keystore and the password
To build this project you have to point the project to the plugins directory of the WSO2 ESB 4.8.1. You may copy the plugins directory somewhere else and point the project to that and build. If you are running the server in Java 7, make sure the plugins directory also contain the patched jars of the above mentioned issue; wss4j, rampart and TicketDecoder.
Now if you run the client you should see the response of the echo service. You should see an output like the following in the console.
The WCF client used to test the echo proxy is located here.
The WCF client creates a form with a button, when clicked it calls the echo proxy. Open App.Config in the Visual Studio.
The client executable is located in EchoWcfClientServiceVisualStudio\EchoClient\bin\Debug
If the executable is not found, you’ll have to build the project in Visual Studio.
Either run the executable or run the client within Visual Studio. This should bring up a form. Click the echo button. The response should be shown in the message box.
You might have noticed that we used the http address. If you want to use the https endpoint for SSL, you have to add the following element in the app.config.
Once the certificates are imported, you should be able to call the proxy with the https endpoint.
When the Key Distribution Center (KDC) is the Active Directory (AD), the SPN is usually attached with a service account. Contact your Windows Administrators to create an SPN and a service account and generate a key tab file against that. The main difference from WSO2 IS and AD based KDC is that, in AD the SPN doesn’t provide a text password, rather it’s in the keyTab file.
To create an SPN for a related service account, use the following command.
setspn -s HTTP/USESB.your.domain.com us\us-wso2esb setspn -s HTTP/USESB us\us-wso2esbChange the SPN and service account accordingly.
Ask the administrators to create the keytab using the following command.
ktpass /out wso2.keytab /princ HTTP/USESB@YOUR.DOMAIN.COM /mapuser us\us-wso2esb /pass password /crypto All /ptype KRB5_NT_PRINCIPALThis will create a keyTab file named wso2.keytab
Get the keytab file and place it in a location accessible by the WSO2 ESB.
In our case the KDC hostname is us.your.domain.com. The ESB configuration is same as like the previous case. But make the following changes in the krb5.conf
[libdefaults]
default_realm = US.YOUR.DOMAIN.COM
default_tgs_enctypes = rc4-hmac des-cbc-crc des-cbc-md5 aes256-sha1 aes128-aha1
default_tkt_enctypes = rc4-hmac des-cbc-crc des-cbc-md5 aes256-sha1 aes128-aha1
permitted_enctypes = rc4-hmac des-cbc-crc des-cbc-md5 aes256-sha1 aes128-aha1
allow_weak_crypto = true
[realms]
US.YOUR.DOMAIN.COM = {
kdc = us.your.domain.com
default_domain = US.YOUR.DOMAIN.COM
}
[domain_realm]
.us.YOUR.DOMAIN.COM = US.YOUR.DOMAIN.COM
us.YOUR.DOMAIN.COM = US.YOUR.DOMAIN.COM
[login]
krb4_convert = true
krb4_get_tickets = false
Have the jaas.conf contents be like the below.Server {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
useTicketCache=false
isInitiator=false
keyTab="/carbon/wso2/esb/keytab/wso2.keytab"
principal="HTTP/USESB@US.YOUR.DOMAIN.COM";
};
Note the useKeyTab and the keyTab property, which is different from the WSO2 IS as a KDC scenario. Provide the path of the keyTab file location to the keyTab parameter.Next when you secure the proxy, you have to provide the SPN name as the AD configured SPN. In our case the SPN is HTTP/USESB@US.YOUR.DOMAIN.COM
Also note since you have no password for the SPN, you can provide the service account password which was created at the AD end. In fact that won’t be used in this case, since the password will get resolved from the keyTab file. Once the proxy is secured, we need to do one more change for Java 7. Go to the proxy dashboard and click Policies. In both, Binding echoSoap11Binding and Binding echoSoap12Binding policies add the following line below the service principal name.
<rampart:property name="kdc.ticket.decoder">org.wso2.carbon.krb.Java7.TicketDecoder</rampart:property>
This is required to be added due to a bug that was identified in the implementation and require a patch from WSO2. If you are a support customer of WSO2, you may open a support ticket to request the patch WSO2-CARBON-PATCH-4.2.0-1811.
If you are not a support customer, you can run the ESB using JRE 1.6.
That’s it in ESB end.
When you start the server, start the ESB with the following system property,
./wso2server.sh -Djava.security.krb5.conf= /carbon/wso2/esb/wso2esb-4.8.1/repository/resources/security/krb5.conf start
You may insert that property in the wso2server.sh file as well.
Now you can try invoking the echo proxy using soap ui and it should fail saying “security header does not found”. This is because the request didn’t contain any security information, such as the kerberos ticket etc. In fact it’s difficult to create a soap ui project to work with kerberos, as the WS-Security spec demands a format for the soap request, such as signing the message payload from kerberos ticket and so on.
We have written a java client to cater this. Clients need to incorporate same mechanism to call a kerberos secured proxy. Please find the Java client here.
Note: however the Java client should be running in a machine where it can connect to the AD based KDC.
Testing with a Java client
Change the krb5.conf of the java client as the following in the java client.
[libdefaults]
default_realm = US.YOUR.DOMAIN.COM
default_tgs_enctypes = rc4-hmac
default_tkt_enctypes = rc4-hmac
permitted_enctypes = rc4-hmac
allow_weak_crypto = true
[realms]
US.YOUR.DOMAIN.COM = {
kdc = us.your.domain.com
}
Note the KDC ip address. Change accordingly. Also change the config.properties to match the windows server.keyStorePath=C:\\Users\\smohamednazeer\\Desktop\\DevKeyTab\\wso2carbon.jks keyStorePassword=wso2carbon axis2ClientPath=C:\\Users\\smohamednazeer\\Documents\\codes\\kerberospocsolution\\kerberosJavaClient\\repo\\conf\\client.axis2.xml policyFilePath=C:\\Users\\smohamednazeer\\Documents\\codes\\kerberospocsolution\\kerberosJavaClient\\repo\\conf\\policy.xml serviceEndpoint=https://<ip-address or domain name>:9443/services/echoIf you are using a custom keystore, use that keystore path and password instead.
The rampart configuration of policy should be changed. Client principal name in this case would be the logged in user. In our case we have the following configurations.
<rampart:RampartConfig xmlns:rampart="http://ws.apache.org/rampart/policy">
<rampart:timestampPrecisionInMilliseconds>true</rampart:timestampPrecisionInMilliseconds>
<rampart:timestampTTL>300</rampart:timestampTTL>
<rampart:timestampMaxSkew>300</rampart:timestampMaxSkew>
<rampart:timestampStrict>false</rampart:timestampStrict>
<rampart:nonceLifeTime>300</rampart:nonceLifeTime>
<rampart:kerberosConfig>
<rampart:property name="client.principal.name">smohamednazeer@US.YOUR.DOMAIN.COM</rampart:property <rampart:property name="client.principal.password">xxxxxxxxxxxx</rampart:property>
<rampart:property name="service.principal.name">HTTP/USESB@US.YOUR.DOMAIN.COM</rampart:property>
<rampart:property name="java.security.krb5.conf">C:\Users\smohamednazeer\Documents\codes\kerberospocsolution\kerberosJavaClient\repo\conf\krb5.conf</rampart:property>
<rampart:property name="java.security.auth.login.config">C:\Users\smohamednazeer\Documents\codes\kerberospocsolution\kerberosJavaClient\repo\conf\jaas.conf</rampart:property>
<rampart:property name="javax.security.auth.useSubjectCredsOnly">true</rampart:property>
<rampart:property name="kdc.ticket.decoder">org.wso2.carbon.krb.Java7.TicketDecoder</rampart:property>
</rampart:kerberosConfig>
</rampart:RampartConfig>
Enter the username you use to login to Windows machine as the client.principal.name, or you have to enter a username that exists in your AD. Enter the password that you use to login to the Windows machine for the client.principal.password, or enter the password of the respective AD user.Note the Java7.TicketDecoder property, if you are running the ESB in Java 7 (Note: To use with Java 7, you'll need to have above mentioned patch applied in the ESB 4.8.1 and ESB 4.9.0) If you are running in Java 6, you may remove that line.
If your server is using a custom keystore (which is normally the case), following needs to be done.
1. Use the same keystore, its alias and password of primary keystore for RegistryKeyStore. This is due to a known issue in the system (see [1]). This will get fixed in the future releases.
[1] https://wso2.org/jira/browse/CARBON-15786
2. Change the config.property of the Java client to use the new keystore and the password
To build this project you have to point the project to the plugins directory of the WSO2 ESB 4.8.1. You may copy the plugins directory somewhere else and point the project to that and build. If you are running the server in Java 7, make sure the plugins directory also contain the patched jars of the above mentioned issue; wss4j, rampart and TicketDecoder.
Now if you run the client you should see the response of the echo service. You should see an output like the following in the console.
Calling service with parameter - Hello Shazni!!!!!!! Request = <p:echoString xmlns:p="http://echo.services.core.carbon.wso2.org"><in>Hello Shazni!!!!!!!</in></p:echoString> The response is : <ns:echoStringResponse xmlns:ns="http://echo.services.core.carbon.wso2.org"><return>Hello Shazni!!!!!!!</return></ns:echoStringResponse>Testing with a Windows Communication Foundation (WCF) client
The WCF client used to test the echo proxy is located here.
The WCF client creates a form with a button, when clicked it calls the echo proxy. Open App.Config in the Visual Studio.
<endpoint address="http://<ip address or domain name>:8280/services/echo" behaviorConfiguration=""
binding="customBinding" bindingConfiguration="customTextMessageEncoding"
contract="WSO2.Echo.echoPortType">
<identity>
<servicePrincipalName value="HTTP/USESB@US.YOUR.DOMAIN.COM" />
</identity>
</endpoint>
Note the endpoint address and the servicePrincipalName.The client executable is located in EchoWcfClientServiceVisualStudio\EchoClient\bin\Debug
If the executable is not found, you’ll have to build the project in Visual Studio.
Either run the executable or run the client within Visual Studio. This should bring up a form. Click the echo button. The response should be shown in the message box.
You might have noticed that we used the http address. If you want to use the https endpoint for SSL, you have to add the following element in the app.config.
<httpsTransport authenticationScheme="Anonymous" bypassProxyOnLocal="false"
hostNameComparisonMode="StrongWildcard" proxyAuthenticationScheme="Anonymous"
realm="" useDefaultWebProxy="true" />
instead of the following http binding,<httpTransport authenticationScheme="Anonymous" bypassProxyOnLocal="false"
hostNameComparisonMode="StrongWildcard" proxyAuthenticationScheme="Anonymous"
realm="" useDefaultWebProxy="true" />And when you run the WCF client in the Windows machine, you would get an SSL error if you haven’t imported the ESB servers' primary keystore’s certificate to the Windows trusted certificates. Assuming you have generated your keystores and exported the certificate out of it, follow the steps here to import the certificate file to the Windows server.
Once the certificates are imported, you should be able to call the proxy with the https endpoint.
Tuesday, July 26, 2016
Importing a certificate in Windows
In Windows start menu,
- open the run window and type “mmc” (without quotes). This will open a console window.
- Click File -> Add/Remove Snap-in.
- Click “Certficates” and click Add.
- Select “Computer account” and click next.
- Click Finish.
- Click certificates and click ok.
- Click “Certificates (Local Computer)”
- Click “Trusted RootCertification Authorities” and then certificates
- Right click Certificates of the Trusted Root Certification Authorities in the left pane and All Tasks -> Import
- Click next and browse your certificate file in the file system and complete the import
Saturday, July 23, 2016
Securing WSO2 ESB proxies with Kerberos - part 1 (WSO2 Identity Server as a KDC)
WSO2 Enterprise Service Bus proxies can be secured in various security mechanisms. There are around 20 out of the box security schemes that we can use to secure proxies. Kerberos is one of them. In this post and next we go through the setup and configuration of the Kerberos with two options.
I would like to thank Prabath for permitting me to copy and modify some of the code he used to demonstrate in his posts.
WSO2 Identity Server (IS) as the Key Distribution Center (KDC)
In this procedure we need to have WSO2 IS and WSO2 ESB setup in two server machines. (You may try this with the same server with two different port offsets for WSO2 IS and WSO2 ESB). It’s assumed that you have configured the two servers beyond this point. Look at ESB and IS documentation for reference at [4] and [5]. WSO2 IS can be used in a standalone mode (without any external database setup) for you to work on this.
In WSO2 IS (here we used IS version 5.0.0), let’s setup the KDC.
First we have to enable the KDC. Open embedded-ldap.xml and enable KDC as below.
Once started the server, we have to create a Service Principal (SPN) and a client principal to use with kerberos ticket granting system (TGS). Once the server is started, navigate to Configure -> Kerberos KDC -> Service Principals and click “Add new Service Principal”. Provide a service principal name, description and a password. In this case we used the following service principal name.
SPN Name : esb/localhost@WSO2.ORG
Now create a new user by navigating to Configure -> Users and Roles -> Users -> Add User.
In our case I have added a user name “test” with a password. This will be the client principal.
That’s all with WSO2 IS KDC configuration.
Let’s now configure security in a sample proxy in WSO2 ESB (We used WSO2 ESB 4.8.1). In my case I’ve secured the default echo proxy. You may do the same for any other proxy.
Configuring ESB for kerberos.
Add the following into jass.conf file and place it in <ESB_HOME>/repository/conf/security directory. Note the principal name configuration
Start the ESB server and navigate to the proxy list.
Click the echo proxy to navigate into the dashboard
Click the security button
Select ‘yes’ in the security section
Select the kerberos security mechanism (Item number 16) and click next
Provide the SPN name you configured in the WSO2 IS together with its password and click Finish
Now you can try invoking the echo proxy using soap ui and it should fail saying “security header does not found”. This is because the request didn’t contain any security information, such as the kerberos ticket etc. In fact it’s difficult to create a soap ui project to work with kerberos, as the specification requires a format for the soap request, such as signing the message payload from kerberos ticket, having a time stamp and so on.
We have written a java client to cater this (Thanks Prabath, once again for permitting me to use and modify your client). Clients need to incorporate same mechanism to call a kerberos secured proxy in the Java code. Please find the client here. We shall discuss about a .Net client in the next post.
Open this project in IntelliJIDEA or your favorite IDE.
Navigate to the config.properties and configure the keyStorePath, keyStorePassword, axis2ClientPath, policyFilePath and servicEndpoint according to your setup. We are using the default keystore in this setup and the configuration is like the following.
Next, open up the policy.xml file and make the rampart configuration changes for client.principal.name, client.principal.password (This should be the user you created in the WSO2 IS), service.principal.name, java.security.krb5.conf, java.security.auth.login.config and javax.security.auth.useSubjectCredsOnly. In my case it’s like the following.
Also open up the krb5.conf file located in the client code and configure your kdc ip address. In this case the ip address of the WSO2 IS server.
Now if you run the client you should see the response of the echo service. You should see an output like the following in the console.
This client contacts the WSO2 IS KDC to get a kerberos ticket and then uses the rampart library to build the request as compliant to the WS-Security specification and send the request to the proxy. At the ESB server end, the rampart resolves the request and wss4j validates the ticket by decryption and validating the signature of the request payload and call the backend service, if those validation passes. Subsequently gets the response and sign it back with the kerberos key and send the response back, once again in compliance with the WS-Security.
That's all the configurations you got to do. This is a very simple example on how to use keberos security scheme in WSO2 ESB with WSO2 IS.
In the next post I'll show how to configure the ESB with AD based KDC (specifically with a keyTab file, instead of using the SPN and it's password as used in this post)
[1] http://wso2.com/library/articles/2012/07/kerberos-authentication-using-wso2-products/
[2] http://blog.facilelogin.com/2010/12/kerberos-authentication-with-wso2-esb.html
[3] https://malalanayake.wordpress.com/2012/12/13/how-to-invoke-the-echo-service-secured-by-kerberos-in-wso2-esb/
[4] https://docs.wso2.com/display/ESB490/WSO2+Enterprise+Service+Bus+Documentation
[5] https://docs.wso2.com/display/IS510/WSO2+Identity+Server+Documentation
- WSO2 Identity Server (IS) as Key Distribution Center (KDC)
- Active directory (AD) as the KDC
I would like to thank Prabath for permitting me to copy and modify some of the code he used to demonstrate in his posts.
WSO2 Identity Server (IS) as the Key Distribution Center (KDC)
In this procedure we need to have WSO2 IS and WSO2 ESB setup in two server machines. (You may try this with the same server with two different port offsets for WSO2 IS and WSO2 ESB). It’s assumed that you have configured the two servers beyond this point. Look at ESB and IS documentation for reference at [4] and [5]. WSO2 IS can be used in a standalone mode (without any external database setup) for you to work on this.
In WSO2 IS (here we used IS version 5.0.0), let’s setup the KDC.
First we have to enable the KDC. Open embedded-ldap.xml and enable KDC as below.
<KDCServer>
<Property name="name">defaultKDC</Property>
<Property name="enabled">true</Property>
<Property name="protocol">UDP</Property>
<Property name="host">localhost</Property>
<Property name="port">${Ports.EmbeddedLDAP.KDCServerPort}</Property>
<Property name="maximumTicketLifeTime">8640000</Property>
<Property name="maximumRenewableLifeTime">604800000</Property>
<Property name="preAuthenticationTimeStampEnabled">false</Property>
</KDCServer>
If you want to change the default realm of the KDC, change the “realm” property. By default it’s WSO2.ORG. We’ll keep it as it’s in this case for simplicity.<Property name="realm">WSO2.ORG</Property>We can also enable the KDC setting in the user-mgt.xml. Enable the following property in the UserStoreManager.
<Property name="kdcEnabled">true</Property>Create a file named jaas.conf with following contents and place inside <IS_HOME>/repository/conf/security directory.
Server {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=false
storeKey=true
useTicketCache=false
isInitiator=false;
};
Client {
com.sun.security.auth.module.Krb5LoginModule required
useTicketCache=false;
};
Create a file name krb5.conf with following contents and place it in <IS_HOME>/repository/conf/security directory. This says your KDC locates in the current machine.[libdefaults]
default_realm = WSO2.ORG
default_tkt_enctypes = rc4-hmac des-cbc-md5
default_tgs_enctypes = rc4-hmac des-cbc-md5
dns_lookup_kdc = true
dns_lookup_realm = false
[realms]
WSO2.ORG = {
kdc = 127.0.0.1
}
Now let’s start the WSO2 IS Server.Once started the server, we have to create a Service Principal (SPN) and a client principal to use with kerberos ticket granting system (TGS). Once the server is started, navigate to Configure -> Kerberos KDC -> Service Principals and click “Add new Service Principal”. Provide a service principal name, description and a password. In this case we used the following service principal name.
SPN Name : esb/localhost@WSO2.ORG
Now create a new user by navigating to Configure -> Users and Roles -> Users -> Add User.
In our case I have added a user name “test” with a password. This will be the client principal.
That’s all with WSO2 IS KDC configuration.
Let’s now configure security in a sample proxy in WSO2 ESB (We used WSO2 ESB 4.8.1). In my case I’ve secured the default echo proxy. You may do the same for any other proxy.
Configuring ESB for kerberos.
Add the following into jass.conf file and place it in <ESB_HOME>/repository/conf/security directory. Note the principal name configuration
Server {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=false
storeKey=true
useTicketCache=true
isInitiator=false
principal="esb/localhost@WSO2.ORG";
};
Add the following to the krb5.conf and place it in in <ESB_HOME>/repository/conf/security directory. Configure the ip address of the WSO2 IS in kdc section. If the WSO2 IS is running in the same server as WSO2 ESB, this is the loop back address 127.0.0.1, which is the case in my scenario. Remember to use all caps for the default_realm.[libdefaults]
default_realm = WSO2.ORG
default_tgs_enctypes = des-cbc-md5
default_tkt_enctypes = des-cbc-md5
permitted_enctypes = des-cbc-md5
allow_weak_crypto = true
[realms]
WSO2.ORG = {
kdc = 127.0.0.1:8000
}
[domain_realm]
.wso2.ORG = WSO2.ORG
wso2.ORG = WSO2.ORG
[login]
krb4_convert = true
krb4_get_tickets = false
Now you can try invoking the echo proxy using soap ui and it should fail saying “security header does not found”. This is because the request didn’t contain any security information, such as the kerberos ticket etc. In fact it’s difficult to create a soap ui project to work with kerberos, as the specification requires a format for the soap request, such as signing the message payload from kerberos ticket, having a time stamp and so on.
We have written a java client to cater this (Thanks Prabath, once again for permitting me to use and modify your client). Clients need to incorporate same mechanism to call a kerberos secured proxy in the Java code. Please find the client here. We shall discuss about a .Net client in the next post.
Open this project in IntelliJIDEA or your favorite IDE.
Navigate to the config.properties and configure the keyStorePath, keyStorePassword, axis2ClientPath, policyFilePath and servicEndpoint according to your setup. We are using the default keystore in this setup and the configuration is like the following.
# Kerberos configs keyStorePath=/home/shazni/ProjectFiles/kerberosJavaClient/repo/resources/wso2carbon.jks keyStorePassword=wso2carbon axis2ClientPath=/home/shazni/ProjectFiles/kerberosJavaClient/repo/conf/client.axis2.xml policyFilePath=/home/shazni/ProjectFiles/kerberosJavaClient/repo/conf/policy.xml serviceEndpoint=https://localhost:9448/services/KerberosTestRegNote my ESB run on port offset of 5 hence the port 9448.
Next, open up the policy.xml file and make the rampart configuration changes for client.principal.name, client.principal.password (This should be the user you created in the WSO2 IS), service.principal.name, java.security.krb5.conf, java.security.auth.login.config and javax.security.auth.useSubjectCredsOnly. In my case it’s like the following.
<rampart:RampartConfig xmlns:rampart="http://ws.apache.org/rampart/policy">
<rampart:timestampPrecisionInMilliseconds>true</rampart:timestampPrecisionInMilliseconds>
<rampart:timestampTTL>300</rampart:timestampTTL>
<rampart:timestampMaxSkew>300</rampart:timestampMaxSkew>
<rampart:timestampStrict>false</rampart:timestampStrict>
<rampart:nonceLifeTime>300</rampart:nonceLifeTime>
<rampart:kerberosConfig>
<rampart:property name="client.principal.name">test_carbon.super</rampart:property>
<rampart:property name="client.principal.password">test123</rampart:property>
<rampart:property name="service.principal.name">esb/localhost@WSO2.ORG</rampart:property>
<rampart:property name="java.security.krb5.conf">repo/conf/krb5.conf</rampart:property>
<rampart:property name="java.security.auth.login.config">repo/conf/jaas.conf</rampart:property>
<rampart:property name="javax.security.auth.useSubjectCredsOnly">true</rampart:property>
</rampart:kerberosConfig>
</rampart:RampartConfig>
Also open up the krb5.conf file located in the client code and configure your kdc ip address. In this case the ip address of the WSO2 IS server.
Now if you run the client you should see the response of the echo service. You should see an output like the following in the console.
Calling service with parameter - Hello Shazni!!!!!!! Request = <p:echoString xmlns:p="http://echo.services.core.carbon.wso2.org"><in>Hello Shazni!!!!!!!</in></p:echoString> The response is : <ns:echoStringResponse xmlns:ns="http://echo.services.core.carbon.wso2.org"><return>Hello Shazni!!!!!
This client contacts the WSO2 IS KDC to get a kerberos ticket and then uses the rampart library to build the request as compliant to the WS-Security specification and send the request to the proxy. At the ESB server end, the rampart resolves the request and wss4j validates the ticket by decryption and validating the signature of the request payload and call the backend service, if those validation passes. Subsequently gets the response and sign it back with the kerberos key and send the response back, once again in compliance with the WS-Security.
That's all the configurations you got to do. This is a very simple example on how to use keberos security scheme in WSO2 ESB with WSO2 IS.
In the next post I'll show how to configure the ESB with AD based KDC (specifically with a keyTab file, instead of using the SPN and it's password as used in this post)
[1] http://wso2.com/library/articles/2012/07/kerberos-authentication-using-wso2-products/
[2] http://blog.facilelogin.com/2010/12/kerberos-authentication-with-wso2-esb.html
[3] https://malalanayake.wordpress.com/2012/12/13/how-to-invoke-the-echo-service-secured-by-kerberos-in-wso2-esb/
[4] https://docs.wso2.com/display/ESB490/WSO2+Enterprise+Service+Bus+Documentation
[5] https://docs.wso2.com/display/IS510/WSO2+Identity+Server+Documentation
Tuesday, July 5, 2016
Unleashing the Git - part 6 - git branching
Branching is a function to launch separate, similar copy of the present workspace for different usage requirements. It's a line of development which exists independent of another, where both share a commom history.
Let's understand branching related commands with an example.
abc
123
def
456
then,
abc
123
def
456
ghi
// Now if you try to switch to master, you won't be allowed until you fix the conflicts.
Now you may decide to keep the xyz change and commit it. What needs to exist, what needs to remove is all up to you. You may even choose remove both. So once you switch to master branch, you won't see the line. Issue the following command in master branch.
Following are few additional commands that is are useful.
As an alternative to 'git pull', we can also use 'git fetch' followed by 'git merge @{u}'
Some more useful commands
Let's understand branching related commands with an example.
$ git init // Execute on a preferred locationAdd a file list.txt and add the following content
abc
123
def
456
then,
$ git add . $ git add -m "Initial commit of list.txt" $ git checkout -b mybranch // Create a new branch called mybranch with exact level of master. $ git checkout -b 64868c9 // Create a new branch called mybranch from the point of the given commit idAbove is equivalent to following two commands.
$ git branch mybranch $ git checkout mybranch $ git branch // Shows available branches with the current branch with an *. Now you'll be in mybranch $ git branch -r // Shows only remote branches. $ git branch -a // Shows all available branches $ git branch --merged // Shows all branches merges into the current branch $ git branch --no-merged // Shows available branches that are not merged with the current branch $ git branch --contains <commit id> // Shows the branch that contains the commit idNow suppose you openned list.txt and modified it by adding a new line, making it
abc
123
def
456
ghi
$ git add . $ git commit -m "Adding a new line in the mybranch" $ git checkout master // Switch back to master. Now list.txt won't have your new line$ git merge mybranch // merges mybranch with the master branch. This would succeed as there was no conflict in lines. We just added a new line, so git would have no problem in merging. Updating 53a7908..3fd44bc Fast-forward list.txt | 1 + 1 file changed, 1 insertion(+) Also git automatically commit while merging. If you do not want the commit to happen, you can use the following command.
$ git merge --no-commit master// Now in the master branch remove the first line and commit it. Then switch to mybranch and open list.txt. You would still see the initial first line we removed in the master. Now modify that line in mybranch to change it to xyz from abc and commit it. Say now you want to merge the files in this branch from master. Issue the following command.
$ git merge master Auto-merging list.txt CONFLICT (content): Merge conflict in list.txt Automatic merge failed; fix conflicts and then commit the result.There will be a conflict as both branches have modified the first line.
// Now if you try to switch to master, you won't be allowed until you fix the conflicts.
$ git checkout master list.txt: needs merge error: you need to resolve your current index firstNow open the list.txt in mybranch. You would see following.
<<<<<<< HEAD xyz ======= >>>>>>> master 123 def 456 ghiWhat does this say? HEAD is pointing to your current branch. i.e. mybranch. Whatever you have between <<<<<<< HEAD to ======= is the the contents of the mybranch offending to the conflict. i.e. your xyz modification. Whatever between ======= and >>>>>>> master is what master has for the offending conflict. In this case it's empty meaning there's an empty line.
Now you may decide to keep the xyz change and commit it. What needs to exist, what needs to remove is all up to you. You may even choose remove both. So once you switch to master branch, you won't see the line. Issue the following command in master branch.
$ git merge mybranch Updating a64cb6d..87fd36d Fast-forward list.txt | 1 + 1 file changed, 1 insertion(+)Now you should see the xyz line in the master branch as well.
Following are few additional commands that is are useful.
$ git merge --log development // Adding a one line log message from each merge commit to the merge message $ git merge --no-ff development // Force creation of a merge commit $ git merge -m "Forced commit message that from the merging branch" mybranchYou can use git commit --amend to modify the commit message after the fact too. Here’s an example:
$ git merge --log --no-ff development $ git commit --amend -c HEAD // Editor launches $ git branch -d mybranch // Deletes the mybranch. If there are unmerged files, git will warn you about it. $ git branch -D mybranch // Git doesn't warn you about unmerged files. It just deletes the branch $ git push origin :beta // Remove a remote origin's branch named beta
$ git push <remote name> <branch name> $ git push origin master:production // Push changes from local master branch to remote origin's production branch $ git push --force or git push -f // To push changes even if you don't have ceratin changes from the remote. Warning: Use this with extreme caution because it can cause others to getout of sync with the repository to which you are pushing.
As an alternative to 'git pull', we can also use 'git fetch' followed by 'git merge @{u}'
Some more useful commands
$ git checkout -b <branch> --track <remote>/<branch> // Checkout a remote branch to your local filesystem $ git pull <remote name> <remote branch> // Pulls changes from the <remote branch> of <remote name> into the current local branch $ git pull <remote name> <remote branch>:<local branch> // Pulls changes from the <remote branch> of <remote name> into the <local branch> // e.g: git pull origin development:team-dev $ git pull --rebase origin master // rebasing the current locak branch on top of remote origin's master branch
Saturday, June 25, 2016
Unleashing the Git - part 5 - git stashing
More often than not, there can be situations where you have done some changes and you need to temporarily hide those from the working tree since those changes are not completely ready yet. Stashing comes into play in such situations. May be when you have done some changes in a branch and you want to switch to another branch without committing your changes in the current branch. Stash is the easiest option for that.
Say you have changed some files. Issuing the git status would show what has changed.
Say you have changed some files. Issuing the git status would show what has changed.
$ git stash // Temporarily hide your changes. Now issue git status. You won't see the modifications. Equivalent to $ git stash save $ git stash save --patch // Opens the text editor to choose what portion to stash $ git stash apply // Apply the last stashed change and keep the stash in stack. But this is normally not done. Instead following is done $ git stash pop // Apply the last stashed change and remove it from stack $ git stash list // Shows available stashes.Stashes have name in the format of stash@{#} where # is 0 for most recent one and 1 for the second recent and so on.
$ git stash drop <stash name> // e.g: git stash drop stash@{0} to remove the first stash
$ git stash clear // Remove all the stashed changes
$ git stash branch <stash name> // Create a branch from an existing stash
In the next part of this series we'll discuss git branching in detail.
Friday, June 24, 2016
Unleashing the Git - part 4 - Git Tagging
Tagging comes very handy in managing a repository. Tags and branches are similar concepts. But tags are read only. There are two types of tags. 1) Lightweight 2) Annotated
$ git tag Basic_Features // Creates a Basic_Features tag. $ git tag // Shows available tags Basic_Features $ git show Basic_Features // Which show the details of the commit on which Basic_Feature tag is added $ git tag beta1 HEAD^ // Creates a tag from next to last commitEarlier I showed how to check out code with commit id. We can also use the tags to checkout code of a particular stage using the following command.
$ git checkout Basic_FeatureAll of the above are lightweight/unannotated tag. Creating an annotated tag is as simple as creating lightweight tag as below,
$ git tag -a Basic_Messaging_Annotated -m "Annotated tag at Messaging" $ git show Basic_Messaging_Annotated // this will provide the details of the tagger as well.Deleting a tag is same for both tag types.
$ git tag -d Basic_Messaging_Annotated // Deletes the annotated tag we created. $ git push origin v1.0 // Push local tag v1.0 to remote origin. $ git push --tags origin // Push all the local tags to remote origin $ git fetch --tags origin // fetches all the remote origins tags to local repository. Note: If you have a local tag with same name as a remote tag, it'll be overwritten
Saturday, June 18, 2016
Unleashing the Git - part 3 - Working with remote repositories
More often we need to share the files that we modify and make it available for other users. This is typically the case in Software development projects, where several people work on a set of files and all needs to modify and do changes. The solution is to have the repository in a centralised place and each and every one work with there own copies before they commit it to the central place. There are many online git repository providers. Examples are GitHub, BitBucket etc.
Choose, whatever best suites for you and sign up for an account. Most online repositories provide free accounts. I've a created a one in bitbucket and have created a repository named online_workbench.
After you entered your password, you'll have your repository online and its local location is your local $HOME/example directory.
'git push -u origin master' is to make the git aware, that all the pull and push operations to default to the master branch. If in case '-u origin master' is not specified, each and every time you issue a 'pull' or 'push' command, you will have to specify the origin.
Ok.. Now if someone needs to get a local copy and work on this remote repository, what he/she needs to do? Only few steps are involved.
First we need to clone the repository into a directory. Navigate or create a new directory where you need the local clone of the repository and issue the following command.
Depending on how the firewall on your computer or local area network (LAN) is configured, you might get an error trying to clone a remote repository over the network. Git uses SSH by default to transfer changes
over the network, but it also uses the Git protocol (signified by having git:// at the beginning of the URI) on port 9418. Check with your local network administrator to make sure communication on ports 22—the port SSH communicates on—and 9418 are open if you have trouble communicating with a remote repository.
Choose, whatever best suites for you and sign up for an account. Most online repositories provide free accounts. I've a created a one in bitbucket and have created a repository named online_workbench.
$ git remote add origin https://shazni@bitbucket.org/shazni/online_workbench.git // Syntax is git remote add <name> <repository URL> $ git remote rm <name> // To remove a remote from your local repository $ git push -u origin master Password for 'https://shazni@bitbucket.org': Counting objects: 4, done. Delta compression using up to 4 threads. Compressing objects: 100% (4/4), done. Writing objects: 100% (4/4), 88.01 KiB, done. Total 4 (delta 0), reused 0 (delta 0) To https://shazni@bitbucket.org/shazni/online_workbench.git * [new branch] master -> masterBranch master set up to track remote branch master from origin.
After you entered your password, you'll have your repository online and its local location is your local $HOME/example directory.
'git push -u origin master' is to make the git aware, that all the pull and push operations to default to the master branch. If in case '-u origin master' is not specified, each and every time you issue a 'pull' or 'push' command, you will have to specify the origin.
Ok.. Now if someone needs to get a local copy and work on this remote repository, what he/she needs to do? Only few steps are involved.
First we need to clone the repository into a directory. Navigate or create a new directory where you need the local clone of the repository and issue the following command.
$ git clone <remote repository location> <Path you want the clone to be>If you omit the <Path you want the clone to be>, it will be in the current directory.
Depending on how the firewall on your computer or local area network (LAN) is configured, you might get an error trying to clone a remote repository over the network. Git uses SSH by default to transfer changes
over the network, but it also uses the Git protocol (signified by having git:// at the beginning of the URI) on port 9418. Check with your local network administrator to make sure communication on ports 22—the port SSH communicates on—and 9418 are open if you have trouble communicating with a remote repository.
$ git clone --depth 50 <remote repository location> // Create a shallow clone with the last fifty commitsThen, if we do the modification to the files or added files, next we need to stage it for commits. Issue the following commands.
$ git add * // Stages and add all your changes, This is similar to git add -A // If you specify git add -u will only previously added and updated files will be staged. -p parameter will present you with sections so that you are given option to include a change or not. $ git commit –m 'My commit comment' // Commit the changes to the local repository $ git pull // checks whether there are any unsinced updates in the remote location and if exist syncs local and the server $ git push // Add the changes made by you to the server
Sunday, February 28, 2016
Building boost C++ library
Boost is a well written portable C++ source library. It's a very popular and flexible library set to use in you C++ applications. It has a huge set of libraries spanning across many utilities and uses. Good thing about Boost is that, the libraries available in Boost are either already available in C++ standard or likely to end up in it one day. Note however, not all libraries are accepted. Nevertheless, all the libraries are proven and well tested.
Download the latest boost library from http://www.boost.org/
In this short guide I'll walk you through the process to easily build boost for you to generate the header and library files for you to use in your C++ application.
Once you download the boost, extract it into a directory. Navigate into the extracted directory and issue the following commands. Make sure you have a C++ compiler available in your system (e.g: g++ in GNU/Linux). Look here (which is a generic post, however) to see how to install C/C++ compilers.
You are done!!! You should be able to find two directories inside the directory you specified in the PREFIX; namely includes and libs. These contain header files and .so (or .dll in Windows) shared library files to link in your C++ application.
Download the latest boost library from http://www.boost.org/
In this short guide I'll walk you through the process to easily build boost for you to generate the header and library files for you to use in your C++ application.
Once you download the boost, extract it into a directory. Navigate into the extracted directory and issue the following commands. Make sure you have a C++ compiler available in your system (e.g: g++ in GNU/Linux). Look here (which is a generic post, however) to see how to install C/C++ compilers.
$ ./bootstrap.sh --prefix=path/to/installation/prefix $ ./b2 installDepending on your system tools, you may not have all the libraries built. For example, if you do not have python dev packages installed, boost python library won't build. But this won't stop building other libraries.
You are done!!! You should be able to find two directories inside the directory you specified in the PREFIX; namely includes and libs. These contain header files and .so (or .dll in Windows) shared library files to link in your C++ application.
Thursday, February 18, 2016
WSO2 Microservices Framework for Java (MSF4J) 1.0.0 is released
WSO2 is pleased to announce that it has released the very first version 1.0.0 of the WSO2 Microservices Framework for Java (MSF4J). You can get more details from the following location.
[1] http://wso2.com/products/microservices-framework-for-java/
It gives you a lightweight fast runtime together with an annotation based programming model to create your micro services. Browse the following location for a quick start guide.
[2] https://docs.wso2.com/display/MSF4J100/Quick+Start+Guide
Enjoy creating your microservices!!!
[1] http://wso2.com/products/microservices-framework-for-java/
It gives you a lightweight fast runtime together with an annotation based programming model to create your micro services. Browse the following location for a quick start guide.
[2] https://docs.wso2.com/display/MSF4J100/Quick+Start+Guide
Enjoy creating your microservices!!!
Saturday, January 30, 2016
Unleashing the Git - part 2 - git log and git stat
This is the 2nd part of the Git posts I'm currently blogging on. You can find the fist part here. This part focuses on the git logging and its various outputs and statistics generation in git. These information are vital for developers. Some of the uses are to see the recent changes, diffs in recent commits, who made what change etc. Let's dive into these commands and see what they have to offer us.
With -e parameter we can get the email address of the user, whereas the -s parameter gives a summary of each users stages/commits.
git log command as I stated earlier gives git commit ID and other meta information. If you specify --skip=<number> it will omit last <number> of commits and show only others. Following are handful of git log examples that you may find useful in your day to day development work.
It's often the case while developing that you add few files to your repository mistakenly or for testing which you never intent to commit. You can use git clean command to remove those. Let's assume we need to remove all the text file with extension .txt
Following two commands will do it.
$ git shortlog
Shazni Nazeer (10):
First commit
Removing compliled files
Removing log files
Verifying folder existence
create directory called logs
Data loading from DB
More logging configurations and message parsing
adding MessageProcessor.py
Handling of few user requests
Bug fixes and Items ID is changed to VARCHAR from INT
Above command arranges all the users who were involved in
building the repository data in alphabetical order, showing number of
commits they have made along with descriptions. Of course if you have
only one user contributing, only that users information is listed. If
you specify a -n parameter to the same command it will order the users
by number of commits. With -e parameter we can get the email address of the user, whereas the -s parameter gives a summary of each users stages/commits.
git log command as I stated earlier gives git commit ID and other meta information. If you specify --skip=<number> it will omit last <number> of commits and show only others. Following are handful of git log examples that you may find useful in your day to day development work.
$ git log -p // Shows the commit changes $ git log -1 -p HEAD // To see the latest commit $ git log -p -2 // Limit the output to only last two commits, also shows the diffs $ git log -5 // Shows only the last 5 commit without diffs. $ git log HEAD^^..HEAD // Same as git log -2 $ git log --skip=4 // Skips the last 4 commit details $ git log --since=2014-02-27 --until=2014-03-10 // Only get commit details between the specified dates $ git log --since=2.days // Find commit of last 2 days - If 3.months - Since three month before $ git log --since="1 week" == git log --after="7 days" == git log --since="168 hours" $ git log --before="1 week" == git log --until="7 days" == git log --before="168 hours" $ git log --author="some user" $ git log --grep="user" // Finds all the commits with a descrpition having the word user. -i paramter at the end ensures case insensitive search $ git log --stat // Gives details with files changed $ git log --pretty=oneline $ git log --pretty=format:"%h - %an, %ar : %s" $ git log --pretty=format:"%h %s" --graph $ git log --oneline // first seven characters of the commit id and the commit message shown $ git log -- some/path/ $ git log -- somefile
It's often the case while developing that you add few files to your repository mistakenly or for testing which you never intent to commit. You can use git clean command to remove those. Let's assume we need to remove all the text file with extension .txt
Following two commands will do it.
$ git clean -f -e*.txt $ git clean -fGetting information in git is not limited to git log command. We often need to know more information on individual files changes and other useful information. git diff command have plethora of options to provide you with required information about your repository. Let's check some of the commands that you might find useful.
$ git diff // The diff between the working tree and the staged area $ git diff --staged // View the difference between staged changes and the repository
View the differences between the working tree and a commit in the repository
$ git diff HEAD $ git diff <Commit ID> $ git diff <first commit id> <second commit id> or $ git diff <first commit id>..<second commit if> // View the differences between two commits. $ git diff -- path/ // Limit the diff output to a specific path $ git diff-tree -p COMMIT // Where COMMIT is your commit number to see what was changed in that commit
Generating stats
git diff can also be used to get some statistics output in addition to differences in the source code. Some of the examples are as shown below.
$ git diff --stat HEAD~5 or git diff --stat HEAD~5 HEAD // Show change stats between last 5 commits $ git diff --stat <first_commit_id> <second_commit_id> // Stats between two commits $ git diff --shortstat HEAD~5 // Shows number of files changed, insertions and deletions. $ git diff --numstat HEAD~5 // Shaows the stat in more $ git diff --stat -p HEAD^ // also shows the oatch in addion to stats $ git log --stat // Shows full stats along with logs $ git log --shortstat // Shows Cumulative stats only along with usual log
There is a command that can used to give information on invidividual user responsible for changes in source file by line number. git blame displays all or a portion of a file with annotations showing when
the change was made, by whom, and, more importantly, in what revision the
change was made.
git blame outputs the following information:
• Short commit ID
• Author’s name
• Date and time of commit
• Line number
$ git blame <some_file> // To show the entire file with line by line info on above $ git blame -L 20 <some_file> // Starts blaming of the beginning from line 20 $ git blame -L 100,120 <some_file> // Limit the blame to lines 100 to 120 $ git blame -L 5,+20 <some_file> // Limit the blame starting from line number 5 to 20 lines $ git blame -L 30,-13 <some_file> // Get the blame of the file from line number 30 to 13 lines behind $ git blame -L "<POSIX Reg Ex>",+20 <some_file> // Get the blame of the specified file from the line matching to regex to 20 lines ahead
Git can track content that moves around in a file or is copied from one file to
another. You can use git blame to show content that has moved around by
adding the -M parameter.
$ git blame -M <some_file> // To see which part has got moved around in the specified file
You can also track changes copied from another file by using the -C
parameter. It checks the changes in the file against other changes in the
repository to see whether it was copied from somewhere else.
$ git blame -C <some_file> // To see which part has got copied from another file in the specified file
Subscribe to:
Posts (Atom)