Ask Experts Questions for FREE Help !
Ask
    priyankamakke's Avatar
    priyankamakke Posts: 1, Reputation: 1
    New Member
     
    #1

    Mar 21, 2019, 02:17 AM
    Execute Apache Spark (Scala) code in Bash script
    I am newbie to spark and scala. I wanted to execute some spark code from inside a bash script. I wrote the following code.
    Scala code was written in a separate [FONT=Consolas, Menlo, Monaco, Lucida Console, Liberation Mono, DejaVu Sans Mono, Bitstream Vera Sans Mono, Courier New, monospace, sans-serif].SCALA[/FONT][FONT=inherit] file as follows.[/FONT]
    Scala Code:
    import org.apache.spark.SparkContext
    import org.apache.spark.SparkContext._
    import org.apache.spark.SparkConf

    object SimpleApp {
    def main(args: Array[String]) {
    val conf = new SparkConf().setAppName("Simple Application")
    val sc = new SparkContext(conf)
    println("x="+args(0),"y="+args(1))
    }
    }This is the bash script that invokes the Apache-spark/scala code.
    Bash Code
    #!/usr/bin/env bash
    Absize=File_size1
    AdBsize=File_size2
    for I in `seq 2 $ABsize`
    do
    for j in `seq 2 $ADsize`
    do
    Abi=`sed -n ""$i"p" < File_Path1`
    Adj=`sed -n ""$j"p" < File_Path2`
    scala SimpleApp.scala $Abi $adj
    done
    doneBut then I get the following errors.
    Errors:
    error:object apache is not a member of package org
    import org.apache.spark.SparkContext
    ^
    error: object apache is not a member of package org
    import org.apache.spark.SparkContext._
    ^
    error: object apache is not a member of package org
    import org.apache.spark.SparkConf
    ^
    error: not found:type SparkConf
    val conf = new SparkConf().setAppName(" < File_Path2`
    scala SimpleApp.scala $Abi $adj
    done
    doneBut then I get the following errors.
    Errors:
    error:object apache is not a member of package org
    import org.apache.spark.SparkContext
    ^
    error: object apache is not a member of package org
    import org.apache.spark.SparkContext._
    ^
    error: object apache is not a member of package org
    import org.apache.spark.SparkConf
    ^
    error: not found:type SparkConf
    val conf = new SparkConf().setAppName(") ^
    error: not found:type SparkContextThe above code works perfectly if the scala file is written without any spark function (That is a pure scala file), but fails when there are apache-spark imports.
    What would be a good way to run and execute this from bash script? Will I have to call spark shell to execute the code?

    bash scala shell apache-spar



    • 3

      Try to provide spark-core dependency in class-path like scala -classpath "*.jar" YourFile.scala – FaigB




    • I used the following command scala -classpath simple-project_2.11-1.0.jar SimpleApp.scala $Abi $adj but still there is a same error @FaigB –aroon


    • 1

      If you have spark locally use in classpath jar from $spark_home/lib/spark-core_2.10-{version}.jar OR download from https://mindmajix.com/apache-spark-training and put on classpath – faigB

    • 1

      Typically, you submit spark jobs with the apark summit command – puhlen

    • 2

      you have 2 choices set up spark with environment variable and run as @puhlen told with spark-submit -class SimpleApp simple-project_2.11-1.0.jar $Abi $adj OR list all required ClassNotFoundException libs in classpath.





    If you want to know more about apache spark (scala) code in bash script then my suggestion is to visit mindmajix for better results.

Check out some similar questions!

Java script code error [ 1 Answers ]

I am trying to do a homework assignment and I am having issues getting the code to work. Here is the task and then my code so far. Any help would be appreciated. Task: A retail company assigns a $5000 store bonus if monthly sales are $100,000 or more. Additionally, if their sales are 125% or...

Onclick JS script to update code based on field input? [ 0 Answers ]

Is there a way to write a JS code that will create an input field and a button. When the button is pressed, it will check the input field and if the correct code is entered into the field, it will update a snippet of code further down? Basically, I have a hosted paypal button set at one price...

Execute immediate 'create table' in script [ 3 Answers ]

My oracle version is 10.2.0.3.0. In Toad I have no problem 1) to create table trg_farinvc as select 2) select * from all_objects where object_name like 'TRG_FARINVC%' 3) drop table TRG_FARINVC PURGE 3) create table trg_farinvc as select --> create table again no problem. 4) drop table...

Calling Java Script Function in C# code [ 1 Answers ]

How to call a java script function in c# code.

CGI for Apache - Premature end of script headers [ 13 Answers ]

I have a post form on my Web site in Common Gateway Interface (CGI) but when people click in order to post it, the message Premature end of script headers: mailform.cgi appears. If you'd like to see what I'm talking about please go to this page : http://www.accesss.net/pnjoindre.html. How can I...


View more questions Search
 

Question Tools Search this Question
Search this Question:

Advanced Search

Add your answer here.