mysql - How to Only select the Columns which can be casted to Numeric Else ignore the column in Spark Sql? -


i have tried lot of things not find proper solution of how select columns can casted double or numeric when have columns of type string. , ignore rest of columns.

suppose have 100 columns of type string , want check columns can casted numeric , select them , ignore other columns.

for example - 

structtype(     structfield(snapshotdate,stringtype,true),      structfield(country,stringtype,true),     structfield(region,stri‌​ngtype,true),     structfield(probabil‌​ity,stringtype,true)‌​,      structfield(bookingamount,stringtype,true),      structfield(revenueamount,stringtype,true) )

here want select revenueamount, probability , bookingamount. other columns either string type or date, ignore them.

is there way that?

thank in advance.

you can use schema method in dataframe @ structtype describe schema of dataframe.

a structtype set of structfields composing dataframe schema, can return seq[structfield] schema (returned schema on dataframe) using seq method.

now can iterate on schema , keeping desired columns using map method.

val df = //your dataframe definition val schema = df.schema val desiredcolumns = schema.seq.map(structfield =>    if (structfield.datatype.simplestring //pass condition) structfiled.name val newdf = df.select(desiredcolumns: _*)

notes :

  • the : _* notation using in scala tell compiler pass values inside , array 1 one (not whole array single parameter)
  • the simplestring method return string representation datatype here's examples (string => string , int => int) note representation start lower case instead of uppercase

Comments

Post a Comment

Popular posts from this blog

python - Best design pattern for collection of objects -

i have following architecture: +-----------------+ | collection | | | | | | | | | +-----------------+ | ^ | | create() | | | | v | getitem(b) +------------+ | | item | | | | | | +------+ | | +------------+ a class managing collection of items, creates items. , these items may need other items collection. the actual code in python, , collection passes parameter created item . see it, bad practice. improvement can see pass few functions of collection needed item , instead of whole instance. for example: class collection: getitem(self, id): ... createitem(self, id): item = item(id, self) # <-- pass self ... class item: __init__(self, id, col): ... col.getite...
Read more

go - serving up pdfs using golang -

Image
just wondering if guys can code. it works serving images not pdf brochure. in advance func main(){ http.handlefunc("/", func(w http.responsewriter, r *http.request){ filename := "/var/www/filedipenser/brochure.pdf" streampdfbytes, err := ioutil.readfile( filename ) if err != nil { fmt.println(err) os.exit(1) } b := bytes.newbuffer(streampdfbytes) w.header().set("content-type", "application/pdf") if _, err := b.writeto(w); err != nil { fmt.fprintf(w, "%s", err) } w.write([]byte("pdf generated")) }) err := http.listenandserve(":4111", nil) if err != nil { log.fatal("listenandserve: ", err) fmt.println(err) } } the code bit inefficient appears working. however, pdf viewers might sensitive extraneous output adding end of http response stream...
Read more

how to add preprocess loader in webpack 2 -

i trying add preprocess loader in webpack 3. have installed successfully. not working here's webconfing file var debug = process.env.node_env !== "production"; var webpack = require('webpack'); var path = require('path'); module.exports = { context: path.join(__dirname, "src"), devtool: debug ? "inline-sourcemap" : null, entry: "./js/app.js", module: { loaders: [ { test: /\.jsx?$/, exclude: /(node_modules|bower_components)/, loader: 'babel-loader!preprocess-loader?+debug', // query: { // presets: ['react', 'es2015'], // plugins: ['react-html-attrs', 'transform-class-properties', 'transform-decorators-legacy'], // } }, { test: /\.css$/, loader: "style-loader!css-loader!sass-loader" } ] }, output: { path: __dirname + "/src/", f...
Read more