Cassandra WriteTimeoutException exception in CounterMutationStage - node dies eventually -


i'm getting following exception in cassandra system.log:

warn  [countermutationstage-25] 2017-07-25 13:25:35,874 abstractlocalawareexecutorservice.java:169 - uncaught exception on thread thread[countermutationstage-25,5,main]: {} java.lang.runtimeexception: org.apache.cassandra.exceptions.writetimeoutexception: operation timed out - received 0 responses.     @ org.apache.cassandra.service.storageproxy$droppablerunnable.run(storageproxy.java:2490) ~[apache-cassandra-3.9.jar:3.9]     @ java.util.concurrent.executors$runnableadapter.call(unknown source) ~[na:1.8.0_112]     @ org.apache.cassandra.concurrent.abstractlocalawareexecutorservice$futuretask.run(abstractlocalawareexecutorservice.java:164) ~[apache-cassandra-3.9.jar:3.9]     @ org.apache.cassandra.concurrent.abstractlocalawareexecutorservice$localsessionfuturetask.run(abstractlocalawareexecutorservice.java:136) [apache-cassandra-3.9.jar:3.9]     @ org.apache.cassandra.concurrent.sepworker.run(sepworker.java:109) [apache-cassandra-3.9.jar:3.9]     @ java.lang.thread.run(unknown source) [na:1.8.0_112] caused by: org.apache.cassandra.exceptions.writetimeoutexception: operation timed out - received 0 responses.     @ org.apache.cassandra.db.countermutation.grabcounterlocks(countermutation.java:150) ~[apache-cassandra-3.9.jar:3.9]     @ org.apache.cassandra.db.countermutation.applycountermutation(countermutation.java:122) ~[apache-cassandra-3.9.jar:3.9]     @ org.apache.cassandra.service.storageproxy$9.runmaythrow(storageproxy.java:1473) ~[apache-cassandra-3.9.jar:3.9]     @ org.apache.cassandra.service.storageproxy$droppablerunnable.run(storageproxy.java:2486) ~[apache-cassandra-3.9.jar:3.9]     ... 5 common frames omitted

whenever happens, cpu goes down 0% minute or so, node becomes unresponsive recovers after that. eventually, node die (i.e. process keeps running, not respond commands more, shutdown not work, have kill process).

some more information:

  • cassandra 3.9
  • g1 garbage collector
  • single node on windows server 2012 r2 (20 cores, 256 gb ram)
  • using lot of counters , counter mutations

things have tried:

  • eleminated other warnings log. used have warnings counter batches being large, rewrote code not use batching @ all. eleminated warning, not exception problem.
  • migrated bigger machine, used bigger heap , fine tuned gc make sure problem not machine being overstressed. cpu load < 20%.

does have idea else do? main concern node dying completely. not sure exception causing hint have...

update 1:

updated cassandra 3.11 , node not seem die more now. however, write timeouts presists, node unresponsive several minutes @ least recovers now.

update 2:

solved problem (with of professional consultant). disc i/o speed on our node terrible, leading growing queue of flush writers. reason unknown, i/o speed tests on drive (raid 1 ssds) super good. moving node windows linux (and configuring according http://docs.datastax.com/en/landing_page/doc/landing_page/recommendedsettings.html) solved problem.

real reason problem unknown; might have been windows per se or freak incompatibility raid setup. in case, cassandra tested on linux , far easier find linux setups. lesson learned.

it sounds beefy machine 20cores , 256gb ram. cassandra distributed system aimed scale horizontally. rather pushing load @ single node, try adding more commodity hardware , scale horizontally. can run multiple nodes of cassandra within same box.

atleast try running couple of nodes within box scale unresponsiveness. cpu not bottleneck cassandra. i/o single node can perform.

  • check values on concurrent_writes in cassandra.yaml, guess based on recommendation 20 cores 160 (20 * 8).
  • if feasible, try separating commitlog directory , data directory storage drives.
  • best bet scale writes add more boxes (which smaller in configuration).

Comments

Post a Comment

Popular posts from this blog

networking - Vagrant-provisioned VirtualBox VM is not reachable from Ubuntu host -

this question exact duplicate of: puphpet/vagrant: unable access lamp vm ip address 1 answer i following basic vagrant tutorial , vagrant provisions box on ubuntu without complaining. moreover, can vagrant ssh vm. however, vm not reachable host: it's not possible ping it's ip address. what have achieved: installed virtualbox (5.1.24 r117012 (qt5.5.1)) , vagrant (1.9.7) onto ubuntu (16.04.2 lts) machine. provisioned vm vagrant up , vagrant finishes job , not complain. vagrant ssh works. i have manually installed lighttpd on guest vm , can ping/curl address: localhost within guest vm. this when trying reach guest vm host: ~/tmp/myfirstvagrantproject$ ping 172.22.22.22 ping 172.22.22.22 (172.22.22.22) 56(84) bytes of data. ... icmp_seq=4 destination net unreachable ... 171 packets transmitted, 0 received, +21 errors, 100% packet loss, time 173
Read more

c# - ASP.NET Core - There is already an object named 'AspNetRoles' in the database -

i'm receiving error applicationdbcontext database when trying update entity framework. i've googled, tried fixes here on so, no avail. i'm deleting database each time make change, once deploy, not option. here's error: pm> update-database -context applicationdbcontext system.data.sqlclient.sqlexception: there object named 'aspnetroles' in database. @ system.data.sqlclient.sqlconnection.onerror(sqlexception exception, boolean breakconnection, action 1 wrapcloseinaction) @ system.data.sqlclient.tdsparser.throwexceptionandwarning(tdsparserstateobject stateobj, boolean callerhasconnectionlock, boolean asyncclose) @ system.data.sqlclient.tdsparser.tryrun(runbehavior runbehavior, sqlcommand cmdhandler, sqldatareader datastream, bulkcopysimpleresultset bulkcopyhandler, tdsparserstateobject stateobj, boolean& dataready) @ system.data.sqlclient.sqlcommand.runexecutenonquerytds(string methodname, boolean async, int32 timeout, boolean
Read more

ruby on rails - ArgumentError: Missing host to link to! Please provide the :host parameter, set default_url_options[:host], or set :only_path to true -

i have built module in solidus , ruby on rails , try create tests. i trying create integration tests , have error: e error: prescriptionphotostest#test_/prescription_photos_post_endpoint_throws_an_error_when_wrong_params: argumenterror: missing host link to! please provide :host parameter, set default_url_options[:host], or set :only_path true test/integration/prescription_photos_test.rb:6:in `block in ' i read this , this , doesn't make difference. my code returning error on line: @user = factorygirl.create(:user) here code files. prescription_photos_rest.rb require 'test_helper' class prescriptionphotostest < actiondispatch::integrationtest setup spree::auth::config[:confirmable] = false @user = factorygirl.create(:user) end def authenticated_header token = knock::authtoken.new(payload: { sub: @user.id }).token { 'authorization': "bearer #{token}" } end test '/prescription_p
Read more