LinZino的筆記

2016年7月19日星期二

scala compiler -classpath with multiple JAR files

1. To compile a scala file and include a Jar file, enter the following command line:

scala  -classpath /home/user/jarFile.jar scalaFile.scala

2. To compile a scala file and include multiple Jar file, enter the following command line:

In Linux OS:
scala  -classpath "/absolute/path/jarFile1.jar:/absolute/path/jarFile2.jar" scalaFile.scala

In Winsows OS:
scala  -classpath "C:\absolute/path/jarFile1.jar;C:/absolute/path/jarFile2.jar" scalaFile.scala

scala 多個jar包編譯

1.scala 編譯時僅有引用一個jar包:

scala  -classpath /home/user/jarFile.jar scalaFile.scala

2. scala 編譯時僅有引用一個jar包:
在Windows底下是分號要注意，Linux則是冒號，也要注意分號跟冒號分隔前後不可以有空白，也要使用絕對路徑，例如:~/xxx/yyy/filejar.jar 是不可以的。

In Linux OS:
scala  -classpath "/absolute/path/jarFile1.jar:/absolute/path/jarFile2.jar" scalaFile.scala

In Winsows OS:
scala  -classpath "C:\absolute/path/jarFile1.jar;C:/absolute/path/jarFile2.jar" scalaFile.scala

2016年6月4日星期六

Scala 錯誤: java.nio.charset.MalformedInputException: Input length = 1

Scala ERROR java.nio.charset.MalformedInputException: Input length = 1

import scala.io.Source

object Test{

def main(args: Array[String] ){

val filename = "test.txt"

Source.fromFile("test.txt", "UTF-8").foreach{

}

在執行上段程式碼的時候，遇到了 java.nio.charset.MalformedInputException: Input length = 1 的錯誤，官方解釋這個錯誤為:

MalformedInputException thrown when an input byte sequence is not legal for given charset, or an input character sequence is not a legal sixteen-bit Unicode sequence.

代表輸入時遇到不合法的字元，所以只要把輸入的字元編碼就可以了。

fromFile 方法有提供編碼的參數:

fromFile (name : java.lang.String, enc : java.lang.String) : Source

creates Source from file with given name, using given encoding, setting its description to filename.

所以改成:

object Test{

def main(args: Array[String] ){

val filename = "test.txt"

Source.fromFile("test.txt", "UTF-8").foreach{

}

2016年5月31日星期二

Scala 學習筆記:function literal

function literal直接拿去翻譯通常會翻做"功能字面"，這個翻譯真的讓人無法理解，所以爬了許多文章後，function literal應該翻做這個"方法的名稱"。
例子1:
scala> (a:Int, b:Int) => a + b

這個function literal 就是 anonymous。也就是這個方法的名稱就是匿名方法。

例子2:

scala> var factor = 3
scala> val multiplier = (i:Int) => i * factor

這個function literal 就是 Closures。也就是這個方法的名稱就是閉包方法。這邊的閉包只是我看到最常見的講法，不是官方確定說法。

參考:
http://stackoverflow.com/questions/12615091/why-is-the-following-scala-function-called-a-closure
http://www.tutorialspoint.com/scala/scala_closures.htm

2016年4月29日星期五

RDMA 範例程式

此程式是由 Writing RDMA applications on Linux Example programs Roland Dreier 這位工程師撰寫的，但是由2007年撰寫的所以在複製程式碼有許多的問題，內部也有一點小bug應該是不小心打錯的，花了一下下午的時間將他補足跟測式執行，為一個由C撰寫兩個數字相加並使用RDMA 傳送的小範例程式。

順便一提，這裡的cc在linux 中就是gcc的意思，可以檢查一下/usr/bin底下的link就可以了。

user @ /usr/bin $ ls -l /usr/bin/cc

lrwxrwxrwx. 1 root root 3 2015-12-08 04:46 /usr/bin/cc -> gcc

編譯前請確認是否有安裝 rdma 跟libibverbs 兩個類別庫，也請確認你的server網卡有支援RDMA。

編譯

client :

$ cc -o client client.c -lrdmacm -libverbs

server :

$ cc -o server server.c -lrdmacm

執行
server :

$ ./server

client : (格式為 client <servername> <val1> <val2>)

$./client 192.168.1.2 123 567

123 + 567 = 690

程式碼:

https://github.com/linzion/RDMA-example-application

參考:
http://www.digitalvampire.org/rdma-tutorial-2007/notes.pdf

ibv_wc defined in <infiniband/verbs.h>.

struct ibv_wc {
        uint64_t                wr_id;          /* ID of the completed Work Request (WR) */
        enum ibv_wc_status      status;         /* Status of the operation */
        enum ibv_wc_opcode      opcode;         /* Operation type specified in the completed WR */
        uint32_t                vendor_err;     /* Vendor error syndrome */
        uint32_t                byte_len;       /* Number of bytes transferred */
        uint32_t                imm_data;       /* Immediate data (in network byte order) */
        uint32_t                qp_num;         /* Local QP number of completed WR */
        uint32_t                src_qp;         /* Source QP number (remote QP number) of completed WR (valid only for UD QPs) */
        int                     wc_flags;       /* Flags of the completed WR */
        uint16_t                pkey_index;     /* P_Key index (valid only for GSI QPs) */
        uint16_t                slid;           /* Source LID */
        uint8_t                 sl;             /* Service Level */
        uint8_t                 dlid_path_bits; /* DLID path bits (not applicable for multicast messages) */
};

c compiler error : 程式中有游離的「」

錯誤：程式中有游離的「\342」
錯誤：程式中有游離的「\210」
錯誤：程式中有游離的「\222」

c compiler 時有時會出現這種錯誤，原因是因為程式碼中含有全形符號並非只有特殊的符號爾以，以上例來說是「－」這個全形dash ，只要換成半形就可以了。

檢查出現在哪裡的方法是:

od -c code.c > log.txt

對照一下就很容易發現是什麼全形符號在作怪了。

參考:
http://blog.csdn.net/sdustliyang/article/details/6851464

2016年4月27日星期三

Linux Nvidia Driver Install

Linux Nvidia Driver Install

在安裝前請確定電腦是否是有獨立顯卡，或者你的電腦是擁有內顯與雙顯卡的筆電，這邊的安裝僅適用於電腦僅有一張獨立顯卡電腦使用，否則會遇到安裝完重登入後會黑畫面也無法進入桌面，但是可以切到其他tty的情況，非單張獨立顯卡請參考此。

1.先去官方網站選擇屬於你的顯卡型號，下載run檔。

wget http://......

2.安裝.run 檔前要先關掉X server，依照不同桌面環境去關掉對應的服務。

GNOME: (Cent OS 通常是這個與 ubuntu 舊版 )

sudo service gdm stop

KDE :

sudo service kdm stop

UNITY: (Ubuntu 14.04 以後 )

sudo service lightdm stop

特殊 : (重開使用 sudo start prefdm)

sudo stop prefdm

3.執行.run 檔，按照步驟即可。

sudo sh NVIDIA-Linux-x86_64-XXX.XX.run

2016年4月20日星期三

spark 1.6.1 Standalone Mode install 最簡易部屬

spark 1.6.1 Standalone Mode install

我們準備兩台 ubuntu 14.04的機器，並在上面建置 spark 1.6.1 Standalone Mode，最簡易部屬，假設第一台電腦 ip為192.168.1.3，第二台電腦 ip為192.168.1.4。

1.從官網下載spark 1.6.1 ，package type 選擇: Pre-built for Hadoop 2.6 and later，點 spark-1.6.1-bin-hadoop2.6.tgz 下載並壓縮。

$ wget http://apache.stu.edu.tw/spark/spark-1.6.1/spark-1.6.1-bin-hadoop2.6.tgz 
$ tar zxvf spark-1.6.1-bin-hadoop2.6.tgz

2. 在每台電腦安裝 java

$ sudo apt-get install openjdk-7-jdk

3. 在每台電腦家目錄編輯 .bashrc，於最後檔案最後加上java路徑。

$vi  ~/.bashrc


....

export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64


export PATH=$JAVA_HOME/bin:$PATH

檢測是否成功:

$java -version

4.在每台電腦編輯

$ vi /etc/hosts


192.168.1.3     ubuntu1


192.168.1.4     ubuntu2

5.建立加密連線不需要密碼

$ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa

$cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

$scp -r  ubuntu@192.168.1.3:~/.ssh ~

6.將ssh連線時詢問關掉。

$sudo vi /etc/ssh/ssh_config

 StrictHostKeyChecking no

為了關掉下面訊息。

ECDSA key fingerprint is 7e:21:58:85:c1:bb:4b:20:c8:60:7f:89:7f:3e:8f:15.
Are you sure you want to continue connecting (yes/no)?

7.設定 spark-env.sh


$ cd  /spark-1.6.1-bin-hadoop2.6/conf


$ mv spark-env.sh.template  spark-env.sh


$ vi spark-env.sh


.....


export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64


.....

7.設定 slaves

$mv slaves.template  slaves

$ vi slaves

#localhost
 ubuntu1
 ubuntu2

8.在spark目錄下啟動Master 跟 slave

$sbin/start-all.sh

9.觀看是否已經啟動。

http://192.168.1.3:8080/

10.執行測試程式


$./bin/spark-submit \

  --class org.apache.spark.examples.SparkPi \

  --master spark://ubuntu1:7077 \

  ./lib/spark-examples-1.6.1-hadoop2.6.0.jar \

  10

以下為用相同過程建置出的環境:

 ✘ paslab@fastnet  ~/zino/spark-1.5.1-bin-hadoop2.6  ./bin/spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master spark://fastnet:7077 \
  --executor-memory 20G \
  --total-executor-cores 100 \
  ./lib/spark-examples-1.5.1-hadoop2.6.0.jar \
  10
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
16/04/22 15:57:06 INFO SparkContext: Running Spark version 1.5.1
16/04/22 15:57:06 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/04/22 15:57:06 INFO SecurityManager: Changing view acls to: paslab
16/04/22 15:57:06 INFO SecurityManager: Changing modify acls to: paslab
16/04/22 15:57:06 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(paslab); users with modify permissions: Set(paslab)
16/04/22 15:57:06 INFO Slf4jLogger: Slf4jLogger started
16/04/22 15:57:06 INFO Remoting: Starting remoting
16/04/22 15:57:06 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.1.2:45824]
16/04/22 15:57:06 INFO Utils: Successfully started service 'sparkDriver' on port 45824.
16/04/22 15:57:06 INFO SparkEnv: Registering MapOutputTracker
16/04/22 15:57:06 INFO SparkEnv: Registering BlockManagerMaster
16/04/22 15:57:06 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-654fc5a5-2c10-4cf9-90d0-aaa46dd6259d
16/04/22 15:57:07 INFO MemoryStore: MemoryStore started with capacity 530.3 MB
16/04/22 15:57:07 INFO HttpFileServer: HTTP File server directory is /tmp/spark-aea15ac1-2dfa-445a-aef4-1859becb1ee6/httpd-73c0986a-57f6-4b0b-9fc4-9f84366393c9
16/04/22 15:57:07 INFO HttpServer: Starting HTTP Server
16/04/22 15:57:07 INFO Utils: Successfully started service 'HTTP file server' on port 35012.
16/04/22 15:57:07 INFO SparkEnv: Registering OutputCommitCoordinator
16/04/22 15:57:07 INFO Utils: Successfully started service 'SparkUI' on port 4040.
16/04/22 15:57:07 INFO SparkUI: Started SparkUI at http://192.168.1.2:4040
16/04/22 15:57:07 INFO SparkContext: Added JAR file:/home/paslab/zino/spark-1.5.1-bin-hadoop2.6/./lib/spark-examples-1.5.1-hadoop2.6.0.jar at http://192.168.1.2:35012/jars/spark-examples-1.5.1-hadoop2.6.0.jar with timestamp 1461311827975
16/04/22 15:57:08 WARN MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.
16/04/22 15:57:08 INFO AppClient$ClientEndpoint: Connecting to master spark://fastnet:7077...
16/04/22 15:57:08 INFO SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20160422155708-0000
16/04/22 15:57:08 INFO AppClient$ClientEndpoint: Executor added: app-20160422155708-0000/0 on worker-20160422155620-192.168.1.2-43391 (192.168.1.2:43391) with 12 cores
16/04/22 15:57:08 INFO SparkDeploySchedulerBackend: Granted executor ID app-20160422155708-0000/0 on hostPort 192.168.1.2:43391 with 12 cores, 20.0 GB RAM
16/04/22 15:57:08 INFO AppClient$ClientEndpoint: Executor added: app-20160422155708-0000/1 on worker-20160422155623-192.168.1.3-47961 (192.168.1.3:47961) with 12 cores
16/04/22 15:57:08 INFO SparkDeploySchedulerBackend: Granted executor ID app-20160422155708-0000/1 on hostPort 192.168.1.3:47961 with 12 cores, 20.0 GB RAM
16/04/22 15:57:08 INFO AppClient$ClientEndpoint: Executor updated: app-20160422155708-0000/0 is now RUNNING
16/04/22 15:57:08 INFO AppClient$ClientEndpoint: Executor updated: app-20160422155708-0000/1 is now RUNNING
16/04/22 15:57:08 INFO AppClient$ClientEndpoint: Executor updated: app-20160422155708-0000/0 is now LOADING
16/04/22 15:57:08 INFO AppClient$ClientEndpoint: Executor updated: app-20160422155708-0000/1 is now LOADING
16/04/22 15:57:08 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 46316.
16/04/22 15:57:08 INFO NettyBlockTransferService: Server created on 46316
16/04/22 15:57:08 INFO BlockManagerMaster: Trying to register BlockManager
16/04/22 15:57:08 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.1.2:46316 with 530.3 MB RAM, BlockManagerId(driver, 192.168.1.2, 46316)
16/04/22 15:57:08 INFO BlockManagerMaster: Registered BlockManager
16/04/22 15:57:08 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
16/04/22 15:57:08 INFO SparkContext: Starting job: reduce at SparkPi.scala:36
16/04/22 15:57:08 INFO DAGScheduler: Got job 0 (reduce at SparkPi.scala:36) with 10 output partitions
16/04/22 15:57:08 INFO DAGScheduler: Final stage: ResultStage 0(reduce at SparkPi.scala:36)
16/04/22 15:57:08 INFO DAGScheduler: Parents of final stage: List()
16/04/22 15:57:08 INFO DAGScheduler: Missing parents: List()
16/04/22 15:57:08 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:32), which has no missing parents
16/04/22 15:57:08 INFO MemoryStore: ensureFreeSpace(1888) called with curMem=0, maxMem=556038881
16/04/22 15:57:08 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 1888.0 B, free 530.3 MB)
16/04/22 15:57:08 INFO MemoryStore: ensureFreeSpace(1202) called with curMem=1888, maxMem=556038881
16/04/22 15:57:08 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1202.0 B, free 530.3 MB)
16/04/22 15:57:08 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.1.2:46316 (size: 1202.0 B, free: 530.3 MB)
16/04/22 15:57:08 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:861
16/04/22 15:57:08 INFO DAGScheduler: Submitting 10 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:32)
16/04/22 15:57:08 INFO TaskSchedulerImpl: Adding task set 0.0 with 10 tasks
16/04/22 15:57:10 INFO SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@192.168.1.2:50829/user/Executor#1430081976]) with ID 0
16/04/22 15:57:10 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, 192.168.1.2, PROCESS_LOCAL, 2161 bytes)
16/04/22 15:57:10 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, 192.168.1.2, PROCESS_LOCAL, 2161 bytes)
16/04/22 15:57:10 INFO TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, 192.168.1.2, PROCESS_LOCAL, 2161 bytes)
16/04/22 15:57:10 INFO TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, 192.168.1.2, PROCESS_LOCAL, 2161 bytes)
16/04/22 15:57:10 INFO TaskSetManager: Starting task 4.0 in stage 0.0 (TID 4, 192.168.1.2, PROCESS_LOCAL, 2161 bytes)
16/04/22 15:57:10 INFO TaskSetManager: Starting task 5.0 in stage 0.0 (TID 5, 192.168.1.2, PROCESS_LOCAL, 2161 bytes)
16/04/22 15:57:10 INFO TaskSetManager: Starting task 6.0 in stage 0.0 (TID 6, 192.168.1.2, PROCESS_LOCAL, 2161 bytes)
16/04/22 15:57:10 INFO TaskSetManager: Starting task 7.0 in stage 0.0 (TID 7, 192.168.1.2, PROCESS_LOCAL, 2161 bytes)
16/04/22 15:57:10 INFO TaskSetManager: Starting task 8.0 in stage 0.0 (TID 8, 192.168.1.2, PROCESS_LOCAL, 2161 bytes)
16/04/22 15:57:10 INFO TaskSetManager: Starting task 9.0 in stage 0.0 (TID 9, 192.168.1.2, PROCESS_LOCAL, 2161 bytes)
16/04/22 15:57:10 INFO SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@192.168.1.3:45907/user/Executor#-348604070]) with ID 1
16/04/22 15:57:10 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.1.2:35234 with 10.4 GB RAM, BlockManagerId(0, 192.168.1.2, 35234)
16/04/22 15:57:10 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.1.3:36667 with 10.4 GB RAM, BlockManagerId(1, 192.168.1.3, 36667)
16/04/22 15:57:11 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.1.2:35234 (size: 1202.0 B, free: 10.4 GB)
16/04/22 15:57:11 INFO TaskSetManager: Finished task 8.0 in stage 0.0 (TID 8) in 1057 ms on 192.168.1.2 (1/10)
16/04/22 15:57:11 INFO TaskSetManager: Finished task 5.0 in stage 0.0 (TID 5) in 1060 ms on 192.168.1.2 (2/10)
16/04/22 15:57:11 INFO TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 1120 ms on 192.168.1.2 (3/10)
16/04/22 15:57:11 INFO TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 1124 ms on 192.168.1.2 (4/10)
16/04/22 15:57:11 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 1161 ms on 192.168.1.2 (5/10)
16/04/22 15:57:11 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 1146 ms on 192.168.1.2 (6/10)
16/04/22 15:57:11 INFO TaskSetManager: Finished task 9.0 in stage 0.0 (TID 9) in 1142 ms on 192.168.1.2 (7/10)
16/04/22 15:57:11 INFO TaskSetManager: Finished task 4.0 in stage 0.0 (TID 4) in 1149 ms on 192.168.1.2 (8/10)
16/04/22 15:57:11 INFO TaskSetManager: Finished task 7.0 in stage 0.0 (TID 7) in 1148 ms on 192.168.1.2 (9/10)
16/04/22 15:57:11 INFO TaskSetManager: Finished task 6.0 in stage 0.0 (TID 6) in 1149 ms on 192.168.1.2 (10/10)
16/04/22 15:57:11 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
16/04/22 15:57:11 INFO DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:36) finished in 2.419 s
16/04/22 15:57:11 INFO DAGScheduler: Job 0 finished: reduce at SparkPi.scala:36, took 2.622159 s
Pi is roughly 3.144648
16/04/22 15:57:11 INFO SparkUI: Stopped Spark web UI at http://192.168.1.2:4040
16/04/22 15:57:11 INFO DAGScheduler: Stopping DAGScheduler
16/04/22 15:57:11 INFO SparkDeploySchedulerBackend: Shutting down all executors
16/04/22 15:57:11 INFO SparkDeploySchedulerBackend: Asking each executor to shut down
16/04/22 15:57:11 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
16/04/22 15:57:11 INFO MemoryStore: MemoryStore cleared
16/04/22 15:57:11 INFO BlockManager: BlockManager stopped
16/04/22 15:57:11 INFO BlockManagerMaster: BlockManagerMaster stopped
16/04/22 15:57:11 INFO SparkContext: Successfully stopped SparkContext
16/04/22 15:57:11 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
16/04/22 15:57:11 INFO ShutdownHookManager: Shutdown hook called
16/04/22 15:57:11 INFO ShutdownHookManager: Deleting directory /tmp/spark-aea15ac1-2dfa-445a-aef4-1859becb1ee6

2016年7月19日 星期二

scala compiler -classpath with multiple JAR files

scala 多個jar包編譯

2016年6月4日 星期六

Scala ERROR java.nio.charset.MalformedInputException: Input length = 1

2016年5月31日 星期二