Zemian Blog: A Programmer's Journal

Tuesday, December 19, 2017

blog: Move yet again

This blog has been moved to https://zemian.github.io/

Monday, October 10, 2016

blog: Move yet again to github.com

One more try.

I have decided to use https://zemian.github.io as my new blogging home.

Friday, July 15, 2016

python: Simple http server with CGI scripts enabled

If you want to experiment some python code as CGI script to serve by a HTTP server, you can get started by these steps:

Create a cgi-bin directory.
Ready!

No, really, it's that simple! Try these CGI scripts out.

Example 1: `cgi-bin/hello.py`

#!/usr/bin/env python3

localvars_table = '<table>'
for x in dir():
  localvars_table += '<tr><td>%s</td></tr>' % x
localvars_table += '</table>'

print("Content-type: text/html")
print("")
print("""<html><body>
<p>Hello World! Your custom CGI script is working. Here are your current Python local variables.</p>
%s
<p>NOTE: If you want to write useful CGI script, try the Python 'cgi' module. See cgitest.py script.</p>
</body></html>""" % (localvars_table))

To test and run this, you simply invoke these couple commands:

bash> chmod a+x cgi-bin/hello.py
bash> python3 -m http.server --cgi

You may now test it on your browser with http://localhost:8000/cgi-bin/hello.py. Hit CTRL+C to stop the server.

If you want to do more with fancy CGI scripts, try the Python's cgi module. Here is another example.

Example 2: `cgi-bin/cgitest.py`

#!/usr/bin/env python3

import cgi
cgi.test()

Again chmod your cgitest.py script and visit http://localhost:8000/cgi-bin/cgitest.py. You will see all the HTTP related data as expected when working with a CGI script. See https://docs.python.org/3/library/cgi.html for more details.

Saturday, July 9, 2016

postgres: How to install posgresql-server with yum on Linux

If you have a RedHat/CentOS/OracleLinux distro of Linux, then yum should be available as your package manager. Here are the notes I have to get PostgreSQL server up running.

bash> yum info postgresql-server
bash> # Verify that's the version you want to install

bash> # Ready to install
bash> sudo su -
bash> yum -y install postgresql-server
bash> service postgresql initdb

bash> # Startup the server manually
bash> service postgresql start

bash> # Make server startup at system reboot
bash> chkconfig postgresql on

bash> # Verify postgres DB is working
bash> su - postgres -c psql
postgres=# \du
postgres=# \q

bash> # We are done, exit root user shell
bash> exit

If you can't find service or chkconfig commands, then check to ensure you have have /sbin in your $PATH.

Friday, July 8, 2016

sudo: How to switch Linux account user without the target user's password

Did you know if you have been granted sudo access to a remote host with su command, then you may switch to any user without the need to type in their password?

Try this out:

zemian@myhost bash> sudo su - postgres
# When prompted for password, enter your own user account password.

# Now you are in as `postgres` user!
postgres@myhost bash>

Or if you want to switch to the root user directly, simply try:

bash> sudo su -

This is very useful when you need to switch to a user account that was only setup just to run applications (eg: postgres, mysql, oracle, or weblogic etc.) and not intented for real user. In this case, you might not even know what the real password is. Above trick should get you switch into that target user account.

Thursday, July 7, 2016

ssh: Login to remote host without password

Most of remote systems are secured by SSH, and to gain remote control with terminal, you would need to ssh into the server. You will be prompted to login with your password on every session. To avoid typing password everytime, you need to setup as authorized client. Here is how you can do that with ssh key.

First on your own client machine, generate the $HOME/.ssh/id_rsa.pub file:

bash> ssh-keygen
# When prompted to enter password, simply hit ENTER key to skip it!
bash> cat  ~/.ssh/id_rsa.pub
xxxyyyzzz zemian@myhost
# You will see a very long string instead of "xxxyyyzzz".

Now you need to copy this public key string into your remote host. You need to ssh into the remote host with your valid password first to setup. If successful, the subsequent ssh into the remote host will not prompt you for password!

bash> ssh myremotehost
# Enter password to gain access

After you are in the remote host:

myremotehost> vim ~/.ssh/authorized_keys
#Paste and append the "xxxyyyzzz" into above file.

If you don’t already have the ~/.ssh/authorized_keys file on remote host, then create it, but ensure you don’t let other users or groups to access it. Use command like this to change the permission:

bash> chmod g-rw,o-rw ~/.ssh/authorized_keys

The cool thing about this is that it affects all ssh related commands, such as scp will now work without prompting you for password!

Have a productive day!

Wednesday, July 6, 2016

wls: How to import SSL cert into WLS DemoTrust.jks keystore file

In WLS, you can import your own SSL cert into it’s trust keystore file to invoke "https" contents. Here is how you do that with the default WLS DemoTrust.jks file.

bash> cd $WL_HOME/server/lib
bash> keytool -keystore DemoTrust.jks -storepass DemoTrustKeyStorePassPhrase -list
bash> keytool -keystore DemoTrust.jks -storepass DemoTrustKeyStorePassPhrase -importcert -alias mycert -file mycert.pem

# Or to delete the entry
bash> keytool -keystore DemoTrust.jks -storepass DemoTrustKeyStorePassPhrase -delete -alias mycert

The file mycert.pem can be obtained by any modern browser when you visit the "https" site. For example using Firefox, you can follow these steps to export the cert file:

Click on the Lock icon next to the URL in the broswer
Click More Information button, then go to the "Security Tab"
Click View Certificate button, then go to the Details tab
Click Export … button
On the bottom right corner dropdown, select X.509 Cerificate with chain (PEM)
Type name of file to save (eg: mycert.pem) and then click Save button

Tuesday, June 28, 2016

pip: How to install venv and pip with a python 3.4 distribution

If you are unfortunate enough to have stuck with a Python 3.4 distribution and you want to setup a virtual env, then likely you will get the ensurepip module not found error:

bash> python3.4 -m venv mypy
Error: Command '['/home/zedeng/py-dev/mypy/bin/python3.4', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1

I found someone posted a nice solution [1] to this. So first remove the dir first.
bash> rm -r mypy

Now do it again like this:
bash> python3.4 -m venv mypy --without-pip
bash> . mypy/bin/activate

mypy> wget https://bootstrap.pypa.io/get-pip.py
mypy> python get-pip.py
mypy> pip list

Now the problem is solved. I believe this problem is gone if you upgrade to Python 3.5

[1] http://www.thefourtheye.in/2014/12/Python-venv-problem-with-ensurepip-in-Ubuntu.html

Monday, June 27, 2016

postgresql: Getting started with PostgreSQL database

I have been learning and using PostgreSQL recently for a Django app, and I have to say it's really GOOD! I have used quite a bit of MySQL and Oracle DB in my workplace, so I didn't find good reason to learn PostgreSQL in the past. Now I finally get a chance to start another new project with Django, so I decided to give it a try, and I wasn't disappointed. I will write up a brief getting started guide here for those impatient to get things moving.

Installing. If you are running on a Mac, the easiest way to try it out is just get the http://postgresapp.com, you just start the App, and your DB is running! No config needed!

After you have the database installed and running, you can either connect to the default "postgres" public database by (psql command), or create your own database. To create your own database, you run this command:

bash> PATH=/Applications/Postgres.app/Contents/Versions/latest/bin:$PATH
bash> createdb mydb

Now you can connect to the new database

bash> psql -d mydb

Once you connected to database, you may run SQL to create/insert/select a table:
mydb-# create table test(id serial, name varchar(200));
mydb-# insert into test(name) values('Hello');
mydb-# select * from test;

To inspect your table columns:
mydb-# \d test;

To see all tables available:
mydb-# \dt;

To see all database available:
mydb-# \l;

To switch to another database:
mydb-# \c mydb2;

To see more help (check out \d options! it's super flexible and useful.):
mydb-# \?;

To quit the psql shell:
mydb-# \q;

To create a database user and set up password, you can do it on bash shell as well:

bash> echo "CREATE ROLE mydbuser1 WITH CREATEDB LOGIN PASSWORD 'Welcome1'" | psql

bash> echo "GRANT ALL PRIVILEGES ON DATABASE mydb TO mydbuser1" | psql

Now you may relogin again with the user. It should prompt you for password.

bash> psql -U mydbuser1 -W

Once you connected to your database again, you may check your user privileges:

mydb-# \dp;

These examples should be enough to get you setup a database for app development. The rest are just normal SQL access to database. The PostgresSQL documentation is excellent reference for all that you need: https://www.postgresql.org/docs/9.5/static/index.html

yum: Installing a local downloaded .rpm file

To install:
bash> sudo yum install <package_name.rpm>

If you have problem installing it due to gpg check failure and you know it's a good RPM, then try with this.
bash> sudo yum --nogpgcheck localinstall <package_name.rpm>

To verify what's in the rpm:
bash> rpm -qlp <package_name.rpm>

hg: Untrack file without deleting it

So I have created a maven based Java project using IntelliJ IDEA, and I added all files to my hg repository with this .hgignore file in place:

syntax: glob
target/

Later I learned that I shouln't track one of IDEA's workspace file, so I want to remove from my repository tracking, but I do not want it be deleted. Here is what you should do:

bash> hg forget .idea/workspace.xml
bash> echo '.idea/workspace.xml' >> .hgignore

Saturday, June 18, 2016

zoom: How to install Zoom conference on Ubuntu 14.04 LTS

1. Download latest zoom_2.0.52458.0531_amd64.deb package from https://zoom.us/download

2. Run
bash> sudo dpkg -i zoom_2.0.52458.0531_amd64.deb

3. Even though the package installed complete, you will see error messages if you have a 64-bit OS because Zoom has other dependencies that's not met. And the above command will setup the apt-get to auto fetch the failed dependencies if you simply run the following next:
bash> sudo apt-get -f install

That's it! Your zoom should be ready to go! Try to start Zoom by press Super + A, then type in "Zoom".

Extra: To Verify Installation, you may run:
bash> dpkg -l |grep zoom

UPDATE:
The exact same steps can be used to install the Skype skype-ubuntu-precise_4.3.0.37-1_i386.deb on Ubuntu.

NOTE: If you have errors running steps above, it's very likely you have upgraded your kernel or added extra repositories into apt-get that's causing all the package dependencies conflict. Try to revert back to original Ubuntu 14.04 install state and these steps should get you running!

Friday, June 17, 2016

wls: How to manually test a DataSource connection

When you creating new DataSource in WLS, it allows you to test the connectivity. However after you have created it, the test button is kinda hidden in a non-obvious way. This is how I get to it and test an existing DataSource that have targeted a server.

1. Login to Admin Console (eg: http://localhost:7100/console)
2. On the Domain Structure left panel, click:
<my_domain_name> (eg: DefaultDomain) > Services > Data Sources
3. On the right, Click on the DataSource name you want to test.
4. Click the "Monitoring" tab > "Testing" tab.
5. Select the server the DataSource has targeted to and then press "Test Data Source" button.

Wednesday, June 15, 2016

hg: How to setup repository on a remote server

I love using mercurial source control! It's easy to use and it works as I expected everytime I need to do something with my source repository. Here is how you can quickly create a remote repository to push your existing project into it. (use this as backup of your project or share among your team etc)

1. SSH into your remote server and create an empty repository.
ssh remote_hostname
bash> cd $HOME
bash> mkdir -p hg-repos/myproject
bash> cd hg-repos/myproject
bash> hg init

2. Back on your local machine and in your existing hg project, edit the .hg/hgrc file with the following:

[paths]
default=ssh://remote_hostname/hg-repos/myproject
[ui]
remotecmd=/usr/local/bin/hg

That's it! You don't even need hg server running in remote host to make this work! Now you may perofrm "hg push" command inside your local project to sync up to remote host

NOTE 1: This setup rely on your SSH into remote host. If you want to avoid user password promt everytime you perform "hg push", then setup Key-Based SSH login.

NOTE 2: You only need "remotecmd" part if your remote_hostname does not have a standard hg installed. Then you can specified your custom location here.

NOTE 3: If you want to use abolute path for "default" path instead, then you can use double slash after the "remote_hostname". Else the single slash is relative to your $HOME directory.

Tuesday, November 3, 2015

Java Jdbc Test for UTF8 and Default Value for TIMESTAMP

Just a quick MySQL test on how to use UTF8 encoding with JDBC connection string. Also a test on how to set TIMESTAMP default values.

package zemian.jdbc;

import org.junit.Test;

import java.sql.*;
import java.util.*;
import java.util.Date;

/**

-- drop table ztest_issues;
CREATE TABLE ztest_issues(
id INT NOT NULL AUTO_INCREMENT PRIMARY KEY
, title VARCHAR(64) NOT NULL
, summary TEXT NULL
, priority INT NOT NULL DEFAULT 5
, cdate DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
);

INSERT INTO ztest_issues(title) VALUES('test');
INSERT INTO ztest_issues(title, summary) VALUES('test2', 'Just a test');
INSERT INTO ztest_issues(title, summary, priority) VALUES('test2', 'Just a test', 1);
INSERT INTO ztest_issues(title, summary, priority, cdate) VALUES('test2', 'Just a test', 1, '2010-12-31 08:00:00');

--INSERT INTO ztest_issues(title, summary) VALUES('locale test1', LOAD_FILE('C:/data/tmp/test.xml'));
--INSERT INTO ztest_issues(title, summary) VALUES('locale test2', LOAD_FILE('C:/data/tmp/test2.xml'));
--INSERT INTO ztest_issues(title, summary) VALUES('locale test3', LOAD_FILE('C:/data/tmp/test3.xml'));

SELECT * FROM ztest_issues;

NOTE:
CURRENT_TIMESTAMP is version specific and is now allowed for DATETIME columns as of version 5.6 http://dev.mysql.com/doc/refman/5.6/en/timestamp-initialization.html

For 5.6 <, use TIMESTAMP for cdate field instead.

cdate TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
cdate DATETIME NULL

NOTE2:
cdate DATETIME NOT NULL DEFAULT 0

If you were to use default to ZERO, then you would need this properties in conn string. Else you will fail to get ZERO date value into Java.

You need to tell the JDBC driver to convert them to NULL. This is done by passing a connection property name zeroDateTimeBehavior with the value convertToNull

For more details see the manual: http://dev.mysql.com/doc/refman/4.1/en/connector-j-installing-upgrading.html

*/
public class ZtestIssuesJdbcTest {
// String url = "jdbc:mysql://localhost/test";
// String url = "jdbc:mysql://localhost/test?zeroDateTimeBehavior=convertToNull";
String url = "jdbc:mysql://localhost/test?useUnicode=true&characterEncoding=UTF-8&connectionCollation=utf8_general_ci&zeroDateTimeBehavior=convertToNull";
String username = "test";
String password = "test123";

@Test
public void testShowTableLocale() throws Exception {
try (Connection conn = DriverManager.getConnection(url, username, password)) {
String sql = "SHOW VARIABLES LIKE 'char%'";
System.out.println("Sql:" + sql);
Statement sm = conn.createStatement();
ResultSet rs = sm.executeQuery(sql);
while (rs.next()) {
System.out.printf("%s\t%s\n", rs.getObject(1), rs.getObject(2));
}

sql = "SHOW CREATE TABLE ztest_issues";
System.out.println("Sql:" + sql);
sm = conn.createStatement();
rs = sm.executeQuery(sql);
while (rs.next()) {
System.out.printf("%s\t%s\n", rs.getObject(1), rs.getObject(2));
}
}
}

@Test
public void testQuery() throws Exception {
try (Connection conn = DriverManager.getConnection(url, username, password)) {
String sql = "SELECT id, cdate, title, summary FROM ztest_issues";
System.out.println("Sql:" + sql);
Statement sm = conn.createStatement();
ResultSet rs = sm.executeQuery(sql);
while (rs.next()) {
System.out.printf("%d\t%s\t%s\t%s\n", rs.getObject(1), rs.getObject(2), rs.getObject(3), rs.getObject(4));
}
}
}

@Test
public void testInsert() throws Exception {
try (Connection conn = DriverManager.getConnection(url, username, password)) {
String sql = "INSERT INTO ztest_issues(title, summary, cdate) VALUES(?, ?, ?)";
System.out.println("Sql: " + sql);
PreparedStatement ps = conn.prepareStatement(sql);

String testId = "" + System.currentTimeMillis();

ps.setObject(1, "English Locale " + testId);
ps.setObject(2, "Just a test");
ps.setObject(3, new Date());
int result = ps.executeUpdate();
System.out.println("Insert Result: " + result);

ps.setObject(1, "Chinese Locale " + testId);
ps.setObject(2, "只是一個測試");
ps.setObject(3, new Date());
result = ps.executeUpdate();
System.out.println("Insert Result: " + result);

ps.setObject(1, "Spanish Locale " + testId);
ps.setObject(2, "Sólo una prueba");
ps.setObject(3, new Date());
result = ps.executeUpdate();
System.out.println("Insert Result: " + result);
}
}
}

Monday, November 2, 2015

ConEmu - a Terminal Windows or Tab Manager

Just learned about ConEmu project at https://conemu.github.io and it's pretty awesome!

Thanks Onur for the tips!

Sunday, October 25, 2015

mac: How to view Unix man pages with browser

I have come across this wonderful project called Bwana (https://www.bruji.com/bwana/) for Mac. Install it and you can view any man page on the browser. For example try typing the following address on the Safari:

man:find

And you will see something like this:

Thursday, October 22, 2015

python: How to setup a new Trac issue tracking system

Here is how I setup a local instance of Trac (a python based issue tracking web application) on my Mac.

Prerequisite: MySQL 5.6 and Python 2.7 (Python3 will not work with Trac yet!)

Step1: Setup a Trac database and a user

sql> CREATE DATABASE trac DEFAULT CHARACTER SET utf8 COLLATE utf8_bin;
sql> CREATE USER 'dev'@'localhost' IDENTIFIED BY 'dev123';
sql> GRANT ALL ON trac.* TO 'dev'@'localhost';

Step2: Install MySQL adaptor for Python and Trac

bash> pip install Genshi trac mysqlclient

Step3: Setup a Track instance

bash> trac-admin /Users/zemian/dev/mytrac intent

bash> # Above will prompt you to enter a backend string. Use
bash> # this connection string:

bash> #   mysql://dev:dev123@localhost:3306/trac

Step3: Create a Trac admin user

bash> htpasswd -c /Users/zemian/dev/mytrac/.htpasswd admin

Step4: Run Track

bash> tracd -p 8000 --basic-auth="mytrac,/Users/zemian/dev/mytrac/.htpasswd,mytrac" /Users/zemian/dev/metric

Saturday, October 17, 2015

python: mysqlclient gives "Library not loaded: libmysqlclient.18.dylib" error

If you want to use python MySQLdb module (eg: if you run Trac with MySQL backend), you would need first install MySQL server on MacOSX, then install the mysqlclient python package using pip. However upon verifying it, you may encounter error like this:

(mypy-test)Zemians-Air:dev zemian$ pip install mysqlclient

Collecting mysqlclient

  Using cached mysqlclient-1.3.6.tar.gz

Building wheels for collected packages: mysqlclient

  Running setup.py bdist_wheel for mysqlclient

  Stored in directory: /Users/zemian/Library/Caches/pip/wheels/9c/3b/73/8f16f45dc76999dafc2af06b0d6e1e669bc0e1594f41fcc2e8

Successfully built mysqlclient

Installing collected packages: mysqlclient

Successfully installed mysqlclient-1.3.6

(mypy-test)Zemians-Air:dev zemian$ python

Python 2.7.10 (default, Aug 22 2015, 20:33:39) 

[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.1)] on darwin

Type "help", "copyright", "credits" or "license" for more information.

>>> import MySQLdb

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File "/Users/zemian/dev/mypy-test/lib/python2.7/site-packages/MySQLdb/__init__.py", line 19, in <module>

    import _mysql

ImportError: dlopen(/Users/zemian/dev/mypy-test/lib/python2.7/site-packages/_mysql.so, 2): Library not loaded: libmysqlclient.18.dylib

  Referenced from: /Users/zemian/dev/mypy-test/lib/python2.7/site-packages/_mysql.so

  Reason: image not found

To resolve this, you need to add the following to your shell environment

export DYLD_LIBRARY_PATH=/usr/local/mysql/lib

Update 2016-06-17
To install mysqlclient on Ubuntu 14, you might need to first install the mysql_config first:
bash> sudo apt-get install libmysqlclient-dev

Monday, October 12, 2015

mysql: How to specify SAME target table for update in FROM clause

Have you tried updating something simple as following?

update category_tmp set last_update=NOW()
where category_id in (
select category_id from category_tmp where name like 'A%'
);

In MySQL you will get an error like this:

Error Code: 1093. You can't specify target table 'category_tmp' for update in FROM clause.

So it says that you can't use the same update TABLE name within the sub query in where condition. The trick to get around this is to use another sub query in the where clause so it won't see the TABLE name being used! Here is a workaround:

update category_tmp set last_update=NOW()
where category_id in (
select category_id from (
select category_id from category_tmp where name like 'A%'
) ID_LIST
);

Noticed that in MySQL, you must name your sub query result such as ID_LIST, in order for it to be re-select it again on the outer query! Otherwise it will error out with this:

Error Code: 1248. Every derived table must have its own alias