Hive for Hadoop - FAILED: Error in metadata: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

Hive for Hadoop

Complete Error -

hive>select count(1) from temp_table;
FAILED: Error in metadata: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

Resolution -

The below steps should help resolve the error -

1. Under hive/bin directory goto metastore_db directory where the hive metadata is stored.

drwxr-xr-x 3 hdd1 root   4096 Jan 15 11:01 ext
drwxr-xr-x 5 hdd1 hddba  4096 Jan 16 11:44 metastore_db
-rw-r--r-- 1 hdd1 hddba 38518 Jan 16 11:44 derby.log
[hdd1@kchjovm41 bin]$ cd metastore_db

2. You should see two .lck files created

[hdd1@kchjovm41 metastore_db]$ ls -ltr
total 48
drwxr-xr-x 2 hdd1 hddba 4096 Jan 16 10:10 log
-rw-r--r-- 1 hdd1 hddba    4 Jan 16 10:10 dbex.lck
-rw-r--r-- 1 hdd1 hddba  857 Jan 16 10:10 service.properties
drwxr-xr-x 2 hdd1 hddba 4096 Jan 16 10:10 seg0
drwxr-xr-x 2 hdd1 hddba 4096 Jan 16 11:44 tmp
-rw-r--r-- 1 hdd1 hddba   38 Jan 16 11:44 db.lck

3. Rename the .lck files

0 [hdd1@kchjovm41 metastore_db]$ mv dbex.lck dbex.lck1

[hdd1@kchjovm41 metastore_db]$ mv db.lck db.lck1
[hdd1@kchjovm41 metastore_db]$

4. Retry the hive sql , should work fine.

Let me know if it helps - Thank you

ssh keys RSA password-less authentication

Quick steps to implement password-less authentication.

Notes:
1. Below steps are for a home environment and hence passphrase is not used.
2. DSA is more secured than RSA.

-On source server - homeserver1

Login to the user account for which  password-less authentication needs to be set , in this case oracle user.


[oracle@homeserver1 .ssh]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_rsa):  ##Hit enter for keys to be placed at default location or mention desired path.
Enter passphrase (empty for no passphrase):  ##Recommended to use a passphrase
Enter same passphrase again:
Your identification has been saved in /home/oracle/.ssh/id_rsa.
Your public key has been saved in /home/oracle/.ssh/id_rsa.pub.
The key fingerprint is:
7b:34:24:c6:f6:76:ad:ea:79:aa:71:81:fd:e5:cd:44 oracle@homeserver1.oracle

## Public and private keys are generated in default path /home/oracle/.ssh

[oracle@homeserver1 .ssh]$ ls -ltr

-rw-r--r--  1 oracle oinstall 235 May  7 00:20 id_rsa.pub

-rw-------  1 oracle oinstall 887 May  7 00:20 id_rsa

-On destination server - homeserver2

Login to the user account for which  password-less authentication needs to be set , in this case oracle user.

Perform same steps as done on homeserver1.

Copy id_rsa.pub generated on homeserver1 as authorized_keys on homeserver2

[oracle@homeserver2 .ssh]$ ls -ltr

total 24

-rw-r--r--  1 oracle oinstall 235 May  7 00:21 id_rsa.pub

-rw-------  1 oracle oinstall 883 May  7 00:21 id_rsa

-rw-r--r--  1 oracle oinstall 235 May  7 00:22 authorized_keys

Similarly copy id_rsa.pub generated on  homeserver2 as authorized_keys on homeserver1.

[oracle@homeserver1 .ssh]$ ls -ltr

-rw-r--r--  1 oracle oinstall 235 May  7 00:21 id_rsa.pub

-rw-------  1 oracle oinstall 883 May  7 00:21 id_rsa

-rw-r--r--  1 oracle oinstall 235 May  7 00:22 authorized_keys

Test the connection both ways..

[oracle@homeserver2 .ssh]$ ssh oracle@homeserver1

Last login: Tue May  7 00:24:37 2013 from homeserver2.oracle

[oracle@homeserver1 ~]$ 

[oracle@homeserver1 .ssh]$ ssh oracle@homeserver2

Last login: Tue May  7 00:23:31 2013 from homeserver1.oracle

[oracle@homeserver2 ~]$

Hence works !

ssh: connect to host port 22: No route to host

I had this error after i had implemented ssh rsa password less authentication( refer my next post for steps to implement ssh rsa password less authentication).

[oracle@homeserver1 .ssh]$ ssh oracle@homeserver2

ssh: connect to host homeserver2 port 22: No route to host

I had firewall enabled which had restriction to port 22 on server homeserver2.

Below are the commands to check the status and turn off firewall.

[root@homeserver2 ~]# /sbin/service iptables stop

Flushing firewall rules:                                   [  OK  ]

Setting chains to policy ACCEPT: filter                    [  OK  ]

Unloading iptables modules:                                [  OK  ]

[root@homeserver2 ~]# /sbin/service iptables status

Firewall is stopped.

After which i was able ssh to my remote server homeserver2 from homeserver1 successfully.

[oracle@homeserver1 .ssh]$ ssh oracle@homeserver2

Last login: Tue May  7 00:23:31 2013 from homeserver1.oracle

[oracle@homeserver2 ~]$

./ggsci: error while loading shared libraries: libnnz10.so: cannot open shared object file: No such file or directory

Yes; this is when you bring up the oracle golden gate console.

Ensure the right oracle home is installed for the Golden Gate version installed.

In this case the golden gate program tries to use libnnz10 that is part of your 10g oracle home.Set manually the $LD_LIBRARY_PATH or set $ORACLE_HOME to point to a 11g oracle home.

libnnz11 > when you using 11g golden gate media.

and libnnz10 > when you are using 10 GG media.

Most importantly make sure your your variable are set right.

Hope it helps.
#################################################################################

ORA-24247: network access denied by access control list (ACL)

Notes : Something new on 11g ..

Oracle allows access to external network services using several PL/SQL APIs (UTL_TCP, UTL_SMTP, UTL_MAIL, UTL_HTTP and UTL_INADDR), all of which are implemented using the TCP protocol. In previous versions of the database, access to external services was effectively an on/off switch based on whether a user was granted execute permissions on a specific package or not. Oracle 11g introduces fine grained access to network services using access control lists (ACL) in the XML DB repository, allowing control over which users access which network resources, regardless of package grants.

##### Run below PROC to setup ACL (run by sys)#############################

create or replace procedure acl_setup(
aacl varchar2,
acomment varchar2,
aprincipal varchar2,
aisgrant boolean,
aprivilege varchar2,
aserver varchar2,
aport number)
is
begin
begin
DBMS_NETWORK_ACL_ADMIN.DROP_ACL(aacl);
dbms_output.put_line('ACL dropped.....');
exception
when others then
dbms_output.put_line('Error dropping ACL: '||aacl);
dbms_output.put_line(sqlerrm);
end;
begin
DBMS_NETWORK_ACL_ADMIN.CREATE_ACL(aacl,acomment,aprincipal,aisgrant,aprivilege);
dbms_output.put_line('ACL created.....');
exception
when others then
dbms_output.put_line('Error creating ACL: '||aacl);
dbms_output.put_line(sqlerrm);
end;
begin
DBMS_NETWORK_ACL_ADMIN.ASSIGN_ACL(aacl,aserver,aport);
dbms_output.put_line('ACL assigned.....');
exception
when others then
dbut.put_line('Error assigning ACL: '||aacl);
dbms_output.put_line(sqlerrm);
end;
commit;
dbms_output.put_line('ACL commited.....');
end;
/
show errors
Procedure created.

####Call Procedure to create new ACL (run by sys)#############################

begin
acl_setup(
'acl_for_apple.xml',
'ACL for apple',
'APPLE',
TRUE,
'connect',
'appleskeepsdocaway.fruits.com',
7777);
end;
/

Where :

acl - The name of the access control list XML file, generated relative to the "/sys/acls" directory in the XML DB Repository.
description - A description of the ACL.
principal - The first user account or role being granted or denied permissions. The text is case sensitive.
is_grant - TRUE to grant, FALSE to deny the privilege.
privilege - Use 'connect' for UTL_TCP, UTL_SMTP, UTL_MAIL and UTL_HTTP access. Use 'resolve' for UTL_INADDR name/IP resolution. The text is case sensitive.
hostname - hostname where links refers to.
Port - port to where destination listens to.

#####PROC to assign new hostname to be run by sys ################
#####Single ACL can access to many hosts #######

BEGIN
DBMS_NETWORK_ACL_ADMIN.ASSIGN_ACL('acl_for_apple.xml', 'havemorefruits.fruits.com',7777);
END;
/
COMMIT;


##################################################################################

ksukia: Attempt to re-kill process OS PID during shutdown immediate

Many have reported this as a bug.
Issue :

Shutdown immediate goes into a wait trying to kill a defunct process.
You should see somthing like this in the alert log.

*** 2011-05-05 04:56:10.282
ksukia: Starting kill, force = 0
ksukia: Attempt 215 to re-kill process OS PID=7135.
ksukia: killed 1 out of 1 processes.
.
.
where PID=7135 is the defunct process.
ps -ef| grep 7135
oraware 7135 7134 0 - ? 0:00
oraware 9921 7878 0 05:25:48 pts/33 0:00 grep 7135
==========================================================================
Resolution

1. Simply exist from the current session ie shutdown immediate.
2. make sure no session on the database (have application down if required).
3. Initiate shutdown immediate again.
4. Should work..!

==========================================================================

ORA-48141: error creating directory during ADR initialization

You probably have it during the startup phase of your database.

Make sure you check or set on the following points :

$ export ORACLE_HOME
$ export PATH
$ export ORACLE_SID

And the entries in the pfile (In my case left a blank line :)