Tuesday, May 22, 2018

Import CMAKE Project to Eclipse CDT

In this post, I will discuss how to import a CMAKE project to Eclipse CDT. Upon Google search, I ran into this solution, but this did not work for me, so here is what I did instead. I am going to make use of cmake-example git repo project to demonstrate how, but you can easily do this for your own.

Open up Eclipse CDT and select File --> New --> C/C++ Project --> C++ Managed Build --> Next. Enter the project name, say cmake-example, and make sure the project type is Empty Project. Also, select the appropriate Toolchains; this will be Linux GCC or MacOSX GCC. Select Finish.

Go to the project root folder, and we will clone the git repository.
$ cd ~/Eclipse/workspace/cmake-example
$ git init
$ git remote add origin https://github.com/bast/cmake-example.git
$ git fetch
$ git checkout -t origin/master

Let's verify that the project compiles.
$ cmake -H. -BDebug
$ cd Debug
$ make -j4

Make sure that the project builds successfully. Also, note down your $PATH environment variable (highlighted in blue below) to be used later. Your variable may differ from mine.
$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin

Now, on Eclipse right-click this project in the Project Explorer pane on the left and select Properties. In the C/C++ Build tab, uncheck both Use default build command and Generate Makefile automatically. Make sure the Build Directory as ${workspace_loc:/cmake-example}/Debug. This is the folder we created with CMAKE in the previous step and built the project. This folder contains CMAKE-generated Makefile, and we are simply asking Eclipse to execute make command in this particular folder.

This is the first step, where the Eclipse simply runs the make command of the Makefile generated from CMAKE. This setup is good if we are not going to edit CMAKE configs anymore. In reality, we probably will need to edit CMAKE configs.

Every time you modify CMakeLists.txt, you will need to re-create Makefile by running the CMAKE command again and again
$ cmake ..

Let's simply automate this build command with Eclipse. This is the second step of this post. Create build.sh in the project folder,
$ vim ~/Eclipse/workspace/cmake-example/build.sh

Simply write down the build commands that you would run from the Debug directory as follows:
cmake ..
make -j4

Now, in the Eclipse open up Project Properties window again. Expand C/C++ Build entry on the left and select Environment. Select Add button, and enter PATH for the Name field and your $PATH environment variable (noted in the previous step) for the Variable field. This is to make sure the Eclipse shell will be able to perform exactly what you can do with your own shell.

In the C/C++ Build tab, enter the following build command in place of make
sh ../build.sh

Select Apply and Close to close the properties window. The project should now successfully build, even if you have modified CMakeLists.txt files.

Lastly, you can modify the run command by selecting Run --> Run Configurations... and browse the executable for C/C++ Application:
~/Eclipse/workspace/cmake-example/Debug/bin/unit_tests

Happy hacking!

Install Ubuntu without a USB Stick

Disclaimer: I recommend that you experiment the method written in this post on a virtual machine first, because it can get quite tricky.

Let's say you want to wipe out your entire system and install Ubuntu. The easiest way is perhaps
1. download Ubuntu Live Image,
2. create a bootable USB stick, and
3. boot from the USB.

Well, if you were like me, who wipe out the entire system often, you will find it quite annoying to locate the USB, create the bootable stick, and so on. Furthermore, what if you don't have a USB stick in possession?

This post is to rescue you in such situations. You can simply download the image and boot from the image stored on your disk! Let's see how we can do this. Some of the references are here and here.

Here is the setup. First, you will need at least two partitions on your disk. One is Linux installation partition, and the other is to hold the iso image. Throughout the post, I am going to assume that your first partition is mounted as / and your second partition is mounted as /data.

You will need to download the Ubuntu Live image to the second partition, say
$ wget http://releases.ubuntu.com/18.04/ubuntu-18.04-desktop-amd64.iso -P /data

Now, you will have the iso image file saved as /data/ubuntu-18.04-desktop-amd64.iso. Make sure that you save the image in the partition other than where the Linux will be installed.

Next, you need to add a grub menu entry.
$ sudo vim /etc/grub.d/40_custom

Your file should look like below:
#!/bin/sh
exec tail -n +3 $0
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.

menuentry "Ubuntu 18.04 LTS" {
set isofile="/ubuntu-18.04-desktop-amd64.iso"
loopback loop (hd0,2)$isofile
echo "Starting $isofile..."
linux (loop)/casper/vmlinuz boot=casper iso-scan/filename=${isofile} quiet splash
initrd (loop)/casper/initrd.lz
}

Let me go over the system partition scheme once more. The above file applies to the partition scheme where
partition 1: /dev/sda1 --> currently mounted as /; will install Linux on this partition
partition 2: /dev/sda2 --> currently mounted as /data; holds the iso image

Since you downloaded the iso image on the /data directory, this is the root of the second partition. Therefore, this is specified as (hd0,2) in the grub menu entry, corresponding to /dev/sda2; we must omit /data here because /data is just the mount-point in currently-running system and grub won't know anything about it. If you have different partition scheme from mine, you will need to edit the entry accordingly.

Finally, you will need to update grub
$ sudo update-grub

Let's reboot the system and see if we can indeed boot from the iso from the current disk.
$ sudo reboot

Make sure to press and hold <Shift> key while booting up, so that grub entry appears. Otherwise, it is likely that grub menu entry won't even appear.

If you have correctly followed till now, you should be able to boot from the iso image. You should even be able to install Ubuntu on partition 1 using the iso image saved in partition 2. However, you will notice that during the installation, it complains that /isodevice cannot be unmounted. You can resolve this issue by running the following in the terminal within the Live Image system (not your currently installed system):
$ sudo umount -l -r -f /isodevice

After running this command, you should be able to successfully wipe out partition 1 and install Ubuntu 18.04 fresh!

Thursday, April 26, 2018

Dimensionality Reduction and Scattered Data Visualization with MNIST

We live in 3D world, and we can only view scattered data in 1D, 2D, or 3D. Yet, we deal with data that have very large dimension. Consider MNIST dataset, which is considered to be a toy example in deep learning field, consists of 28 X 28 gray images; that is 784 dimensions.

How would this MNIST data look like in 2D or 3D after dimensionality reduction? Let's figure it out! I am going to write the code in Pytorch. I have to say, Pytorch is so much better than other deep learning libraries, such as Theano or Tensorflow. Of course, it is just my personal opinion, so let's not get into this argument here.

What I want to do is to take Pytorch's MNIST example found here, and make some modifications to reduce the data dimension to 2D and plot scattered data. This will be a very good example that shows how to do all the following in Pytorch:
1. Create a custom network
2. Create a custom layer
3. Transfer learning from an existing model
4. Save and load a model

Here is the plot I get from running the code below.


This code is tested on Pytorch 0.3.1.

Wednesday, April 25, 2018

Three Minutes Daily Vim Tip: Highlight Current Cursor Line

In Vim, it is often difficult to locate the current cursor, especially when you jump around. Here is a tip to easily locate the cursor.

Simply add the following line to ~/.vimrc
set cursorline

That's it!

Python IDE with YouCompleteMe

OK, I have tried YouCompleteMe, but then I stopped using it after some time. Well, I am going to give it a try once more. The reason is that I am usually developing on a server through ssh. I could launch GUI-based IDE, such as PyCharm through X-forwarding, but the response is too slow. I realized that editing with Vim on a remote server is probably the best option here.

In any case, I explained how to install YouCompleteMe plugin in my previous post. Here, I will talk about a few more useful tips.

One of the must-have feature in IDE is perhaps GoTo feature, where one can go to the definition of certain method of variable. YouCompleteMe also provides this feature. Formally, this is mapped with
:YcmCompleter GoTo

We can create a shortcut. Add the following in the ~/.vimrc file
nnoremap <C-P> :YcmCompleter GoTo<CR>

In the source file that YouCompleteMe understands, you can simply move the cursor to the symbol and press <Control> + p key to go to the definition. To go back, you can press <Control> + o key.

Also, add the following to ~/.vimrc file
let g:ycm_python_binary_path = 'python'

This will load the default python first found on PATH environment.

Happy coding!

Tuesday, April 17, 2018

Android Camera2 Bare Minimum Code Sample

Google has replaced camera API with camera2 API a few years ago. I guess it is a good thing that Google is working hard to improve Android, personally I think camera2 API is just so difficult. Compared to the deprecated camera API, it requires so much more code.

I am trying to understand this new camera2 API, and it is just not easy. I looked at Google's Camera2Basic sample code, it is still daunting for Android beginners like myself. Well, here is my attempt to trim down this fat sample code into the bare minimum so that it is easier to understand the fundamentals of the camera2 API.

All it does is to connect to a camera and display its preview to the screen. No capture, no manual focus, no error checks, etc. I am not happy that even this bare-bone app requires so many lines. Anyways, I will add more functionalities in the upcoming post.



Sunday, April 15, 2018

OpenCV on PyCharm

PyCharm is such a nice IDE for developing in Python. I really love it. However, with OpenCV module, I realized that PyCharm didn't recognize cv2 module.

After some search, here is a very simple solution! Credit goes to here.
When you import cv2, simply copy paste the whole thing below:

import cv2

# this is just to unconfuse pycharm
try:
    from cv2 import cv2
except ImportError:
    pass

That's it!

Friday, April 13, 2018

Redirecting Outputs from Parallel Subprocess Runs

Say you are call some executable from python in parallel. You don't want to clog the stdout, so you want to save each stdout from executable as a file. Below is what you can do.

Saturday, March 17, 2018

Solution to Android CountDownTimer Skipping the Last Tick

Google's implementation of CountDownTimer in Android is a bit strange; it often skips the very last tick. I can see why they did it this way: they want to make sure that onFinish() function will be called with no delay in the case onTick() takes so much time. However, this implantation causes the last tick to skip pretty much.

Well, here is my modified version of CountDownTimer which will not skip the last tick, unless onTick() function takes too long, which it shouldn't. So, make sure that onTick() will return quickly first.

The code modification starts at line 114 below:
The original source code is taken from here.

Saturday, February 3, 2018

Prevent Process Termination over SSH Connection

When running heavy long-time running processes on a remote server through ssh, it is quite annoying that your processes are terminated when you are disconnected. Here is a tip just for this.

Let's assume you ssh into your server
$ ssh USERNAME@SERVER_IP

You want to run some program, which will run for quite some time.
$ ./a.out arg1 arg2 ...

What happens if your ssh connection is lost, perhaps due to unstable network connection, etc. Well, your program a.out will terminate itself, so you will have to re-run it. In fact, you will need to keep your ssh connection until the program completes, which is quite annoying.

One solution is using nohup
$ nohup ./a.out arg1 arg2 ... &

This will redirect the output to nohup.out file. With this, even if you disconnect from ssh connection, your program will keep running in the background.

Another solution, which is probably much more sophisticated, is to use screen. It is your work session where you can run processes, and you can easily detach from it to do other things while your processes running inside the session are intact and not terminated. You can always re-attach to the screen session and resume your processes still running.

To use this, you need to first create a new screen session
$ ssh USERNAME@SERVER_IP
$ screen -S SESSION_NAME

This will create a new screen. Here, you can run your program
$ ./a.out arg1 arg2 ...

Next, you can always detach from this current session with <CTRL + a> d. That is, press <CTRL> and <a> keys together, release the keys and then press <d> key and release. This will detach from the session. At this point, your program will keep running regardless of whether you disconnect from your ssh connection to the server.

To list running screen sessions, run
$ screen -list

To re-attach to the session, you simply run
$ screen -r SESSION_NAME

To exit the screen, you run
$ exit

By the way, you may notice that within screen session, your scroll will register as up/down arrow keys. If you want your scroll to work as scroll terminal output, you can run the following (credit to pistos)
$ echo 'termcapinfo xterm* ti@:te@' >> ~/.screenrc

Happy hacking!

Friday, February 2, 2018

Three Minutes Daily Vim Tip: Display Indentation Levels

It could be just me, but it is difficult for me to sometimes tell the indentation level of multiple indented lines. So here is a tip for someone like me:

IndentLine is an excellent vim plugin that does it for you. For installation, simply use Vundle; insert the following line in your .vimrc
Plugin 'Yggdroot/indentLine'

If you are not too familiar with Vundle, refer to this post where I explained how to use Vundle to install a different plugin.

Thursday, February 1, 2018

Debugging Bash Scripts

Bash scripts are extremely handy when mostly dealing with Linux commands, such as find, grep, sed, and so on. One can write functions, just like in any other languages. However, I have to admit that I haven't really dealt with Bash scripting much so far. One of the reasons, I suppose, is that it is difficult to debug Bash scripts. I didn't think there was any debugging tool for Bash. Well, I just realized that I was wrong! There is in fact a very nice debugger for Bash: bashdb

To install, you can run the following on macOS
$ brew install bashdb

or the following for Ubuntu
$ sudo apt-get install bashdb

To start debugging, run
$ bashdb bash_script.sh argument1 argument2 ...

Many of the commands are similar to that of gdb, but for more info, type in help. Also, this documentation can be of great reference.

Happy bashing!

Wednesday, January 17, 2018

Replace Lines with Given Strings in Shell

Let's learn how to replace given lines by given string. Consider, for example,
$ cat some_file
abcdefg
hijklmn
opqrstu
vwxyz
12345
67890

Let's replace line 2 with string THIS_IS_NEW_LINE2
$ sed '2s/.*/THIS_IS_NEW_LINE2/g' some_file
abcdefg
THIS_IS_NEW_LINE2
opqrstu
vwxyz
12345
67890

You can replace multiple lines with multiple -e options
$ sed -e '2s/.*/THIS_IS_NEW_LINE2/g' -e '3s/.*/THIS_IS_NEW_LINE3/g' some_file
abcdefg
THIS_IS_NEW_LINE2
THIS_IS_NEW_LINE3
vwxyz
12345
67890

Of course, you can always use -i option to modify file on the fly
$ sed -i '2s/.*/THIS_IS_NEW_LINE2/g' some_file
$ cat some_file
abcdefg
THIS_IS_NEW_LINE2
opqrstu
vwxyz
12345
67890

Print Selected Lines in Shell

Let's assume you want to print certain lines in given text. For example, consider
$ cat some_file
1
2
3
4
5
6
7
8
9

Let's say you want to print lines 3-5. This is very easy using sed:
$ sed -n '3,5p' some_file
3
4
5

Let's say you want to print all lines, except 3-5. This can be done with d option:
$ sed '3,5d' some_file
1
2
6
7
8
9

Note that by default sed prints all lines, so -n option asks sed to print only 3 to 5 lines and suppresses the default behavior. On the other hand, the d letter in the single quotes asks sed to delete the lines 3-5 from printing by default, so it prints only the rest of the lines instead. 

Find Line Number Matching Given Expression in Shell

Say you want to find the line number that matches a given search string. For instance,
$ cat some_file.txt
this is some file line 1
it was written in Vim line 2
let's learn unix commands line 3

To find an expression, say unix, we can use grep
$ grep unix some_file
let's learn unix commands line 3

To show the line number of the expression, use -n option
$ grep -n unix some_file
3:let's learn unix commands line 3

To output the line number only, pipe with cut command
$ grep -n unix some_file | cut -d : -f 1
3

Here, -d option specifies delimiter, and -f option specifies the position separated by the delimiter.

If there are multiple lines matching the given expression, you can always use -m option of grep to print just first N of them
$ grep -n -m 2 unix some_file | cut -d : -f 1
1
2

Happy hacking!

Monday, January 15, 2018

Build HTK on macOS with Minimum Effort

In the last post, I presented instructions to build HTK on Ubuntu. In this post, I will show you how to compile HTK on macOS.

As before, download HTK latest stable version source code, 3.4.1 at the time of writing, from here. Extract the files as usual.
$ tar xfz HTK-3.4.1.tar.gz
$ cd htk

Next, you need some required software tools from Apple.
$ xcode-select --install

Now, let's configure and make
$ ./configure --prefix=$(pwd)
$ make

You will probably encounter an error complaining
(cd HTKLib && /Applications/Xcode.app/Contents/Developer/usr/bin/make HTKLib.a) \

 || case "" in *k*) fail=yes;; *) exit 1;; esac;

gcc  -ansi -g -O2 -DNO_AUDIO -D'ARCH="darwin"' -I/usr/include/malloc -Wall -Wno-switch -g -O2 -I. -DPHNALG   -c -o HGraf.o HGraf.c
HGraf.c:73:10: fatal error: 'X11/Xlib.h' file not found

There are two options here. The first is to simply skip HSLab package that requires X11, thereby building without X11.
$ ./configure --prefix=$(pwd) --disable-hslab
$ make
$ make install

You will see all binary files in the bin folder.


The other option is to install X11 library. Download XQuartz from here and install it. This will create /opt/X11/ folder with necessary library and header files. We can now manually compile HTKLib package by specifying the header files
$ cd HTKLib
$ gcc  -ansi -g -O2 -DNO_AUDIO -D'ARCH="darwin"' -I/usr/include/malloc -Wall -Wno-switch -g -O2 -I. -DPHNALG   -c -o HGraf.o HGraf.c -I /opt/X11/include
$ cd ..

We also need to set library path:
$ export LIBRARY_PATH=/opt/X11/lib

Now, we are ready to resume the make process
$ make
$ make install

Viola! You should see bin directory with all HTK binary files!

Saturday, January 13, 2018

Build HTK in Ubuntu 16.04 with Minimum Effort

This post will walk through compilation of HTK on Ubuntu 16.04 64-bit. For macOS, take a look at this post.

First, you will need to download HTK from here. In this post, I will assume you download the latest stable version, 3.4.1. Extract the source files and change the directory
$ tar xfz HTK-3.4.1.tar.gz && cd htk

Because HTK was designed for 32-bit OS, we need to set the environment as such for 64-bit OS. Enter 32-bit environment by running
$ linux32 bash

Now, we do the usual things, starting off with configure
$ ./configure --prefix=$(pwd) && make

You will see an error:
HGraf.c:73:77: fatal error: X11/Xlib.h: No such file or directory
compilation terminated.

To find out which library you need, you can run the following
$ sudo apt-get install apt-file -y && apt-file update
$ apt-file search Xlib.h

You will see some lines of search result, one of which should read
libx11-dev: /usr/include/X11/Xlib.h

From this, you know that you need to install libx11-dev package.
$ sudo apt-get install libx11-dev -y

Let's resume make
$ make

You will encounter yet another error, complaining
Makefile:77: *** missing separator (did you mean TAB instead of 8 spaces?).  Stop.

It turns out that there is an error in HLMTools/Makefile line 77. In the very beginning of the line, there are a number of spaces, which should be replaced by a single tab.
<INSERT TAB HERE>if [ ! -d $(bindir) -a X_ = X_yes ] ; then mkdir -p $(bindir) ; fi

OK, let's continue.
$ make

You should see the compilation succeeds without any problem. Let's install this and you should find HTK binary files inside bin folder in the current directory!
$ make install
$ bin/HCopy

How to SSH into VirtualBox Guest from Host with NAT

Suppose you want to install Linux on your Macbook with VirtualBox. Since GUI is so heavy and resources-consuming, you may want to run Linux guest system in the headless mode and ssh into the guest system from your Macbook. The following shall let you do so.

1. Select Guest system in VirtualBox menu open up settings.

2. In the Network tab, make sure you have enabled an adapter with NAT.

3. Open up Advanced menu and click Port Forwarding.

4. Create a rule where Protocol = TCP, Host IP = 127.0.0.1, Host Port = 2222, Guest IP = 10.0.2.15, and Guest Port = 22.

5. Now, install openssh-server on your guest system, and you should be good to go by running the command from your host:
$ ssh -p 2222 guest_user@127.0.0.1