solr - DeltaImport fetches all the data -


i'm indexing data database. i'm using delta import fetch updated data. however, find fetching whole data twice , processing once though changes applicable 1 row.

my config.xml deltaquery given:

<dataconfig> <datasource type="jdbcdatasource" driver="com.github.cassandra.jdbc.cassandradriver" url="jdbc:c*://127.0.0.1:9042/test" autocommit="true" rowlimit = '-1' batchsize="-1"/> <document name="content">     <entity name="test" query="select * person" deltaimportquery="select * person seq=${dataimporter.delta.seq}" deltaquery="select seq person last_modified > '${dataimporter.last_index_time}' allow filtering" autocommit="true">         <field column="seq" name="id" />         <field column="last" name="last_s" />         <field column="first" name="first_s" />         <field column="city" name="city_s" />         <field column="zip" name="zip_s" />         <field column="street" name="street_s" />         <field column="age" name="age_s" />         <field column="state" name="state_s" />         <field column="dollar" name="dollar_s" />         <field column="pick" name="pick_s" />     </entity> </document> </dataconfig>

there 2100000 rows. cause large memory consumption resulting in running out of memory. problem? or work in way only?

if solr running out of memory time add more memory solr box. adding more ram alleviate issue.


Comments

Post a Comment

Popular posts from this blog

python - Best design pattern for collection of objects -

i have following architecture: +-----------------+ | collection | | | | | | | | | +-----------------+ | ^ | | create() | | | | v | getitem(b) +------------+ | | item | | | | | | +------+ | | +------------+ a class managing collection of items, creates items. , these items may need other items collection. the actual code in python, , collection passes parameter created item . see it, bad practice. improvement can see pass few functions of collection needed item , instead of whole instance. for example: class collection: getitem(self, id): ... createitem(self, id): item = item(id, self) # <-- pass self ... class item: __init__(self, id, col): ... col.getite...
Read more

go - serving up pdfs using golang -

Image
just wondering if guys can code. it works serving images not pdf brochure. in advance func main(){ http.handlefunc("/", func(w http.responsewriter, r *http.request){ filename := "/var/www/filedipenser/brochure.pdf" streampdfbytes, err := ioutil.readfile( filename ) if err != nil { fmt.println(err) os.exit(1) } b := bytes.newbuffer(streampdfbytes) w.header().set("content-type", "application/pdf") if _, err := b.writeto(w); err != nil { fmt.fprintf(w, "%s", err) } w.write([]byte("pdf generated")) }) err := http.listenandserve(":4111", nil) if err != nil { log.fatal("listenandserve: ", err) fmt.println(err) } } the code bit inefficient appears working. however, pdf viewers might sensitive extraneous output adding end of http response stream...
Read more

sharepoint online - C# CSOM SPView ListItemCollection did not update after I add new field in the SP view -

spclientcontext.credentials = new sharepointonlinecredentials(username, securepassword); //https://msdn.microsoft.com/en-us/library/office/ee534956(v=office.14).aspx web spweb = spclientcontext.web; list splist_csv = spweb.lists.getbytitle("supplier master list"); view spview_csv = splist_csv.views.getbytitle("_data pending export csv");//oracle export field spclientcontext.load(spview_csv); spclientcontext.executequery(); camlquery query = new camlquery(); query.viewxml = spview_csv.viewquery; listitemcollection items = splist_csv.getitems(query); spclientcontext.load(items); spclientcontext.executequery(); foreach (var item in items) { //some field/item missing? } hi, using code above retrieve data sharepoint online. @ first, working fine. but, when add in new column view _data pending export csv on sharepoint online, found out new added field missing. idea? ok. found out others issue, lookup field on limit. did fix change lookup field s...
Read more