java - Union of sets in spark querying cassandra -


the table structure in cassandra: 

identifier, date, set(integer)

what want achieve using spark grouping rows identifier , date, , aggregating sets value. more clear example:

raw data: (consider letters representing integers)

id1, 05-05-2017, {a,b,c} id1, 05-05-2017, {c,d} id1, 26-05-2017, {a,b,c} id1, 26-05-2017, {b,c} id2, 26-05-2017, {a,b,c} id2, 26-05-2017, {b,c,d}

output:

id1, 05-05-2017, {a,b,c,d} id1, 26-05-2017, {a,b,c} id2, 26-05-2017, {a,b,c,d}

since set, want unique values in aggregated results. using java , dataset.

if dataframe has columns mentions can this:

df.withcolumn("set", explode(col("set"))).groupby("identifier", "date").agg(collect_set("set"))

Comments

Post a Comment

Popular posts from this blog

python - Best design pattern for collection of objects -

i have following architecture: +-----------------+ | collection | | | | | | | | | +-----------------+ | ^ | | create() | | | | v | getitem(b) +------------+ | | item | | | | | | +------+ | | +------------+ a class managing collection of items, creates items. , these items may need other items collection. the actual code in python, , collection passes parameter created item . see it, bad practice. improvement can see pass few functions of collection needed item , instead of whole instance. for example: class collection: getitem(self, id): ... createitem(self, id): item = item(id, self) # <-- pass self ... class item: __init__(self, id, col): ... col.getite...
Read more

go - serving up pdfs using golang -

Image
just wondering if guys can code. it works serving images not pdf brochure. in advance func main(){ http.handlefunc("/", func(w http.responsewriter, r *http.request){ filename := "/var/www/filedipenser/brochure.pdf" streampdfbytes, err := ioutil.readfile( filename ) if err != nil { fmt.println(err) os.exit(1) } b := bytes.newbuffer(streampdfbytes) w.header().set("content-type", "application/pdf") if _, err := b.writeto(w); err != nil { fmt.fprintf(w, "%s", err) } w.write([]byte("pdf generated")) }) err := http.listenandserve(":4111", nil) if err != nil { log.fatal("listenandserve: ", err) fmt.println(err) } } the code bit inefficient appears working. however, pdf viewers might sensitive extraneous output adding end of http response stream...
Read more

python - django admin: changing the way a field (w/ relationship to another model) is submitted on a form so that it can be submitted multiple times -

Image
struggling find exact wording need use images. i have model called statements , want change way admin page editing or submitting statement looks. particularly field statement has called statement-contexts . right now, doesn't accurately represent field supposed , doesn't accomodate multiple entries. statement-contexts draws many-to-many relationship model called context . context consists of 2 fields pair 2 words: word field , keyword field draws many many relationship model keyword , pool of keywords in database. this drop down menu doesn't accomodate submitting pair of words that's supposed comprise statement-contexts , beyond selecting 2 words (which hope indicate context 's word field , keyword field respectively. it isn't user friendly since people have submit many pairs of contexts , keywords. more 1 drop down menu needed, , way see many different pairs of words need add. so need figure out how change interface. i looking @ creating...
Read more