javagroups-users Mailing List for JGroups
Brought to you by:
belaban
You can subscribe to this list here.
| 2001 |
Jan
|
Feb
|
Mar
|
Apr
|
May
(1) |
Jun
(3) |
Jul
|
Aug
|
Sep
(4) |
Oct
(2) |
Nov
(3) |
Dec
|
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2002 |
Jan
|
Feb
(4) |
Mar
(1) |
Apr
|
May
(4) |
Jun
(13) |
Jul
(8) |
Aug
(12) |
Sep
(21) |
Oct
(47) |
Nov
(33) |
Dec
(29) |
| 2003 |
Jan
(38) |
Feb
(44) |
Mar
(32) |
Apr
(48) |
May
(30) |
Jun
(24) |
Jul
(70) |
Aug
(89) |
Sep
(58) |
Oct
(25) |
Nov
(42) |
Dec
(53) |
| 2004 |
Jan
(64) |
Feb
(74) |
Mar
(18) |
Apr
(72) |
May
(22) |
Jun
(40) |
Jul
(66) |
Aug
(44) |
Sep
(23) |
Oct
(47) |
Nov
(96) |
Dec
(52) |
| 2005 |
Jan
(35) |
Feb
(74) |
Mar
(52) |
Apr
(43) |
May
(74) |
Jun
(60) |
Jul
(39) |
Aug
(51) |
Sep
(60) |
Oct
(57) |
Nov
(90) |
Dec
(82) |
| 2006 |
Jan
(74) |
Feb
(84) |
Mar
(92) |
Apr
(127) |
May
(139) |
Jun
(58) |
Jul
(47) |
Aug
(42) |
Sep
(68) |
Oct
(86) |
Nov
(76) |
Dec
(73) |
| 2007 |
Jan
(38) |
Feb
(42) |
Mar
(50) |
Apr
(51) |
May
(70) |
Jun
(80) |
Jul
(69) |
Aug
(131) |
Sep
(57) |
Oct
(90) |
Nov
(148) |
Dec
(75) |
| 2008 |
Jan
(125) |
Feb
(136) |
Mar
(92) |
Apr
(94) |
May
(44) |
Jun
(83) |
Jul
(35) |
Aug
(52) |
Sep
(91) |
Oct
(129) |
Nov
(129) |
Dec
(48) |
| 2009 |
Jan
(74) |
Feb
(59) |
Mar
(76) |
Apr
(76) |
May
(56) |
Jun
(117) |
Jul
(83) |
Aug
(62) |
Sep
(61) |
Oct
(129) |
Nov
(97) |
Dec
(84) |
| 2010 |
Jan
(56) |
Feb
(93) |
Mar
(80) |
Apr
(49) |
May
(37) |
Jun
(106) |
Jul
(71) |
Aug
(65) |
Sep
(146) |
Oct
(70) |
Nov
(80) |
Dec
(40) |
| 2011 |
Jan
(98) |
Feb
(83) |
Mar
(132) |
Apr
(58) |
May
(45) |
Jun
(55) |
Jul
(58) |
Aug
(68) |
Sep
(59) |
Oct
(26) |
Nov
(88) |
Dec
(31) |
| 2012 |
Jan
(57) |
Feb
(103) |
Mar
(85) |
Apr
(40) |
May
(44) |
Jun
(54) |
Jul
(25) |
Aug
(24) |
Sep
(10) |
Oct
(25) |
Nov
(61) |
Dec
(25) |
| 2013 |
Jan
(34) |
Feb
(52) |
Mar
(16) |
Apr
(61) |
May
(44) |
Jun
(45) |
Jul
(74) |
Aug
(59) |
Sep
(38) |
Oct
(37) |
Nov
(53) |
Dec
(16) |
| 2014 |
Jan
(14) |
Feb
(46) |
Mar
(38) |
Apr
(13) |
May
(67) |
Jun
(31) |
Jul
(45) |
Aug
(12) |
Sep
(13) |
Oct
(14) |
Nov
(52) |
Dec
(26) |
| 2015 |
Jan
(34) |
Feb
(36) |
Mar
(29) |
Apr
(16) |
May
(14) |
Jun
(41) |
Jul
(22) |
Aug
(28) |
Sep
(26) |
Oct
(42) |
Nov
(54) |
Dec
(85) |
| 2016 |
Jan
(39) |
Feb
(9) |
Mar
(42) |
Apr
(39) |
May
(25) |
Jun
(33) |
Jul
(20) |
Aug
(12) |
Sep
(2) |
Oct
(8) |
Nov
(8) |
Dec
(12) |
| 2017 |
Jan
(5) |
Feb
(29) |
Mar
(16) |
Apr
(5) |
May
(8) |
Jun
(9) |
Jul
(19) |
Aug
(9) |
Sep
(6) |
Oct
(23) |
Nov
(15) |
Dec
(3) |
| 2018 |
Jan
(1) |
Feb
(1) |
Mar
(3) |
Apr
(10) |
May
(14) |
Jun
(16) |
Jul
(1) |
Aug
(8) |
Sep
(1) |
Oct
(26) |
Nov
(12) |
Dec
(6) |
| 2019 |
Jan
(3) |
Feb
(2) |
Mar
(5) |
Apr
(5) |
May
(14) |
Jun
(1) |
Jul
(1) |
Aug
|
Sep
(5) |
Oct
|
Nov
|
Dec
(1) |
| 2020 |
Jan
(20) |
Feb
(3) |
Mar
(6) |
Apr
(15) |
May
(2) |
Jun
|
Jul
(5) |
Aug
(5) |
Sep
(1) |
Oct
|
Nov
(5) |
Dec
|
| 2021 |
Jan
|
Feb
|
Mar
|
Apr
(2) |
May
(8) |
Jun
|
Jul
|
Aug
|
Sep
(1) |
Oct
|
Nov
(3) |
Dec
|
| 2022 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
(4) |
Jul
|
Aug
|
Sep
|
Oct
(4) |
Nov
(1) |
Dec
|
| 2023 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
(1) |
Oct
|
Nov
|
Dec
|
| 2026 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
(1) |
Oct
(4) |
Nov
(7) |
Dec
|
|
From: Questions/problems r. to u. J. <jav...@li...> - 2026-11-05 12:55:05
|
Hi again, I'm not sure which combination of logging out/into things got me back to the issues list, but I have filed https://redhat.atlassian.net/browse/JGRP-3052 with a project and steps for reproducing it. Thanks, Bobby |
|
From: Questions/problems r. to u. J. <jav...@li...> - 2026-11-05 10:51:19
|
On 04.11.2026 19:21, Questions/problems related to using JGroups via javagroups-users wrote: > On Wed, Nov 4, 2026 at 6:56 AM Questions/problems related to using > JGroups via javagroups-users <jav...@li...> > wrote: > > Sure, go ahead. Concise instruction how to reproduce this and > possibly a reproducer would be good to include... > > > Hi, thanks. I have a simple test case (single class and pom.xml to > package everything) that can show the behavior. But I can't get to any > place to file a bug. > > I've tried https://issues.redhat.com/browse/JGRP and it redirects me > to https://redhat.atlassian.net/servicedesk/customer/portals and I > cannot find the JGRP issues list. The first link above works for me, even without having signed in. Perhaps you need to clear your cache/cookies for redhat.com? > I got there at some point and created an account, but now that I have > logged in every place I go drops me back at the general customer > portal page. I can't even view issues, e.g. > https://redhat.atlassian.net/browse/JGRP-2818 picked at random, > because I'm redirected to the customer portal page. This links also works for me, both with or without having signed in! Anybody else having issues with accessing JIRA? > Do you know how I can get to some place to file the issue? > Alternatively I can drop a .zip of the maven project here along with a > description of the output I see. > > Thanks, > Bobby > > > > > > _______________________________________________ > javagroups-users mailing list > jav...@li... > https://lists.sourceforge.net/lists/listinfo/javagroups-users -- Bela Ban |http://www.jgroups.org |
|
From: Questions/problems r. to u. J. <jav...@li...> - 2026-11-04 20:47:07
|
I won't bother you with details, but working on my access to file.... On Wed, Nov 4, 2026 at 1:27 PM Questions/problems related to using JGroups via javagroups-users <jav...@li...> wrote: > On Wed, Nov 4, 2026 at 6:56 AM Questions/problems related to using JGroups > via javagroups-users <jav...@li...> wrote: > >> Sure, go ahead. Concise instruction how to reproduce this and possibly a >> reproducer would be good to include... >> > > Hi, thanks. I have a simple test case (single class and pom.xml to package > everything) that can show the behavior. But I can't get to any place to > file a bug. > > I've tried https://issues.redhat.com/browse/JGRP and it redirects me to > https://redhat.atlassian.net/servicedesk/customer/portals and I cannot > find the JGRP issues list. I got there at some point and created an > account, but now that I have logged in every place I go drops me back at > the general customer portal page. I can't even view issues, e.g. > https://redhat.atlassian.net/browse/JGRP-2818 picked at random, because > I'm redirected to the customer portal page. > > Do you know how I can get to some place to file the issue? Alternatively I > can drop a .zip of the maven project here along with a description of the > output I see. > > Thanks, > Bobby > > > > _______________________________________________ > javagroups-users mailing list > jav...@li... > https://lists.sourceforge.net/lists/listinfo/javagroups-users > |
|
From: Questions/problems r. to u. J. <jav...@li...> - 2026-11-04 18:21:58
|
On Wed, Nov 4, 2026 at 6:56 AM Questions/problems related to using JGroups via javagroups-users <jav...@li...> wrote: > Sure, go ahead. Concise instruction how to reproduce this and possibly a > reproducer would be good to include... > Hi, thanks. I have a simple test case (single class and pom.xml to package everything) that can show the behavior. But I can't get to any place to file a bug. I've tried https://issues.redhat.com/browse/JGRP and it redirects me to https://redhat.atlassian.net/servicedesk/customer/portals and I cannot find the JGRP issues list. I got there at some point and created an account, but now that I have logged in every place I go drops me back at the general customer portal page. I can't even view issues, e.g. https://redhat.atlassian.net/browse/JGRP-2818 picked at random, because I'm redirected to the customer portal page. Do you know how I can get to some place to file the issue? Alternatively I can drop a .zip of the maven project here along with a description of the output I see. Thanks, Bobby |
|
From: Questions/problems r. to u. J. <jav...@li...> - 2026-11-04 11:56:05
|
Sure, go ahead. Concise instruction how to reproduce this and possibly a reproducer would be good to include... On 04.11.2026 12:52, Questions/problems related to using JGroups via javagroups-users wrote: > Hi there, > > Could I file a bug and you can give it a look later? The customer who > came up with this case (after an outage) would like to at least know > why it's happening, if not have it fixed. Putting the behavior another > way: if node3 is the one being isolated, why does node2 see node1 > disappear? Or, half the time, node1 sees node2 disappear. > > Thanks, > Bobby > > > On Tue, Nov 3, 2026 at 5:38 AM Questions/problems related to using > JGroups via javagroups-users <jav...@li...> > wrote: > > Hi Bobby > > I suspect that this is an old view change that was retransmitted > until the members who were still in the view received it. If a > member was *not* part of the view anymore, it would have rejected > that view message. > > Again, this is only a suspicion. We could check this with a > reproducer, but then again, I have other stuff on my todo list, > and this seems to be an edge case. > Shout if you disagree... :-) > Cheers > > On 22.10.2026 13:07, Questions/problems related to using JGroups > via javagroups-users wrote: >> Hi again, >> >> Thanks for all of this. I have another ask about it: can you give >> a brief explanation of why the view changes work out the way they >> do? This is from the original email: >> >> --- start --- >> The test case isolates node3 for 20s, reconnects it again and >> waits 60s, then isolates it for 30s and reconnects it again. A >> few seconds >> after that last reconnection, there is a view change for node3 >> and another >> node (sometimes node1, sometimes node2), excluding the third >> node. That >> third node doesn't see a view change until later when there is a >> merge of >> all three again. Since the odd view change happens when all three >> nodes are >> available to each other, the customer would like to know why this is >> happening. >> --- end --- >> >> Listing out the view changes: >> >> 1) node3/node2 see a new view of just those two nodes. node1 does >> not see a view change. (sometimes it's nodes 3/1 that see a >> change and 2 does not) >> 2) all three nodes see a merge view change containing all three. >> >> In other words, why does node3 and another node see a view change >> of just the two of them? >> >> Thanks, >> Bobby >> >> >> On Thu, Oct 1, 2026 at 11:06 AM Questions/problems related to >> using JGroups via javagroups-users >> <jav...@li...> wrote: >> >> tcp_retries2=3 will likely work, but JGroups cannot handle >> the underlying issue alone: if no packets are received at the >> (TCP/)IP level, then all bets are off. >> >> For example, if no heartbeats are received, a member (or >> multiple members) will get suspected and excluded from the >> cluster. However, due to the same issue, the ensuing view >> will not get delivered to all members until tcp_retries2 >> kicks in. >> >> This creates all sorts of issues that JGroups (and no other >> software) cannot fix. >> >> On 01.10.2026 15:45, Questions/problems related to using >> JGroups via javagroups-users wrote: >>> On Thu, Oct 1, 2026 at 7:36 AM Questions/problems related to >>> using JGroups via javagroups-users >>> <jav...@li...> wrote: >>> >>> [...] >>> 5.5.7? Super! Since this was released 3 weeks ago, is >>> this already part of a product? >>> >>> >>> Our next release will be in a couple months and will include >>> 5.5.7. The customer had an earlier version, but I bumped up >>> to the latest before looking at this test, reproduced the >>> same thing there, etc etc. I just didn't want to make my >>> email any longer than it already was. :) >>> >>> >>> I'm wondering how the iptables test case relates to >>> their actual problem. Could it be a firewall / switch issue? >>> >>> >>> I don't think we'll know exactly what happened there, but >>> this customer was interested enough to put in the time to >>> come up with a way to reproduce the unexpected (to us) view >>> changes. They had one of the three nodes on a different >>> subnet, though there was no technical reason for it. I've >>> suggested they put everything in one place, and >>> hopefully they won't have this kind of issue again. >>> >>> Thank you for looking at this. I really just wanted to make >>> sure that the TCP explanation wasn't a Claude hallucination >>> that just happened to work out when I fiddled with the TCP >>> settings. >>> >>> >>> >>> Your customer needs to evaluate this in terms of the >>> tradeoff between quick detection of failures >>> (tcp_retries2=3) and premature closing of a TCP >>> connection (this takes longer with tcp_retries2=15). For >>> example, Oracle RAC recommends a value of 3. >>> >>> The question is: is this an extreme edge case? Then I'd >>> leave the tcp_retries2 value at 15, else reduce it. >>> Setting tcp_retries2 affects the entire host, not just >>> an individual process, so think about the consequences >>> for other applications. >>> >>> >>> I think this is just a case of "why does this happen?" I >>> will tell them that they can reduce tcp_retries2 if it's >>> important to them to handle this test case, but really they >>> should leave it alone and get the nodes into a network that >>> doesn't have this kind of intermittent disconnection. Even >>> if JGroups can handle it, my product sitting on top still >>> needs to communicate between the nodes fairly regularly. >>> >>> Cheers, and thanks as always for the quick help, >>> Bobby >>> >>> >>> >>> >>> _______________________________________________ >>> javagroups-users mailing list >>> jav...@li... >>> https://lists.sourceforge.net/lists/listinfo/javagroups-users >> >> -- >> Bela Ban |http://www.jgroups.org >> >> _______________________________________________ >> javagroups-users mailing list >> jav...@li... >> https://lists.sourceforge.net/lists/listinfo/javagroups-users >> >> >> >> _______________________________________________ >> javagroups-users mailing list >> jav...@li... >> https://lists.sourceforge.net/lists/listinfo/javagroups-users > > -- > Bela Ban |http://www.jgroups.org > > _______________________________________________ > javagroups-users mailing list > jav...@li... > https://lists.sourceforge.net/lists/listinfo/javagroups-users > > > > _______________________________________________ > javagroups-users mailing list > jav...@li... > https://lists.sourceforge.net/lists/listinfo/javagroups-users -- Bela Ban |http://www.jgroups.org |
|
From: Questions/problems r. to u. J. <jav...@li...> - 2026-11-04 11:52:57
|
Hi there, Could I file a bug and you can give it a look later? The customer who came up with this case (after an outage) would like to at least know why it's happening, if not have it fixed. Putting the behavior another way: if node3 is the one being isolated, why does node2 see node1 disappear? Or, half the time, node1 sees node2 disappear. Thanks, Bobby On Tue, Nov 3, 2026 at 5:38 AM Questions/problems related to using JGroups via javagroups-users <jav...@li...> wrote: > Hi Bobby > > I suspect that this is an old view change that was retransmitted until the > members who were still in the view received it. If a member was *not* part > of the view anymore, it would have rejected that view message. > > Again, this is only a suspicion. We could check this with a reproducer, > but then again, I have other stuff on my todo list, and this seems to be an > edge case. > Shout if you disagree... :-) > Cheers > > On 22.10.2026 13:07, Questions/problems related to using JGroups via > javagroups-users wrote: > > Hi again, > > Thanks for all of this. I have another ask about it: can you give a brief > explanation of why the view changes work out the way they do? This is from > the original email: > > --- start --- > The test case isolates node3 for 20s, reconnects it again and > waits 60s, then isolates it for 30s and reconnects it again. A few seconds > after that last reconnection, there is a view change for node3 and another > node (sometimes node1, sometimes node2), excluding the third node. That > third node doesn't see a view change until later when there is a merge of > all three again. Since the odd view change happens when all three nodes are > available to each other, the customer would like to know why this is > happening. > --- end --- > > Listing out the view changes: > > 1) node3/node2 see a new view of just those two nodes. node1 does not see > a view change. (sometimes it's nodes 3/1 that see a change and 2 does not) > 2) all three nodes see a merge view change containing all three. > > In other words, why does node3 and another node see a view change of just > the two of them? > > Thanks, > Bobby > > > On Thu, Oct 1, 2026 at 11:06 AM Questions/problems related to using > JGroups via javagroups-users <jav...@li...> > wrote: > >> tcp_retries2=3 will likely work, but JGroups cannot handle the underlying >> issue alone: if no packets are received at the (TCP/)IP level, then all >> bets are off. >> >> For example, if no heartbeats are received, a member (or multiple >> members) will get suspected and excluded from the cluster. However, due to >> the same issue, the ensuing view will not get delivered to all members >> until tcp_retries2 kicks in. >> >> This creates all sorts of issues that JGroups (and no other software) >> cannot fix. >> >> On 01.10.2026 15:45, Questions/problems related to using JGroups via >> javagroups-users wrote: >> >> On Thu, Oct 1, 2026 at 7:36 AM Questions/problems related to using >> JGroups via javagroups-users <jav...@li...> >> wrote: >> >>> [...] >>> 5.5.7? Super! Since this was released 3 weeks ago, is this already part >>> of a product? >>> >> >> Our next release will be in a couple months and will include 5.5.7. The >> customer had an earlier version, but I bumped up to the latest before >> looking at this test, reproduced the same thing there, etc etc. I just >> didn't want to make my email any longer than it already was. :) >> >> >>> >>> I'm wondering how the iptables test case relates to their actual >>> problem. Could it be a firewall / switch issue? >>> >> >> I don't think we'll know exactly what happened there, but this customer >> was interested enough to put in the time to come up with a way to reproduce >> the unexpected (to us) view changes. They had one of the three nodes on a >> different subnet, though there was no technical reason for it. I've >> suggested they put everything in one place, and hopefully they won't have >> this kind of issue again. >> >> Thank you for looking at this. I really just wanted to make sure that the >> TCP explanation wasn't a Claude hallucination that just happened to work >> out when I fiddled with the TCP settings. >> >> >> >>> Your customer needs to evaluate this in terms of the tradeoff between >>> quick detection of failures (tcp_retries2=3) and premature closing of a TCP >>> connection (this takes longer with tcp_retries2=15). For example, Oracle >>> RAC recommends a value of 3. >>> >>> The question is: is this an extreme edge case? Then I'd leave the >>> tcp_retries2 value at 15, else reduce it. Setting tcp_retries2 affects the >>> entire host, not just an individual process, so think about the >>> consequences for other applications. >>> >> >> I think this is just a case of "why does this happen?" I will tell them >> that they can reduce tcp_retries2 if it's important to them to handle this >> test case, but really they should leave it alone and get the nodes into a >> network that doesn't have this kind of intermittent disconnection. Even if >> JGroups can handle it, my product sitting on top still needs to communicate >> between the nodes fairly regularly. >> >> Cheers, and thanks as always for the quick help, >> Bobby >> >> >> >> >> _______________________________________________ >> javagroups-users mailing lis...@li...://lists.sourceforge.net/lists/listinfo/javagroups-users >> >> >> -- >> Bela Ban | http://www.jgroups.org >> >> _______________________________________________ >> javagroups-users mailing list >> jav...@li... >> https://lists.sourceforge.net/lists/listinfo/javagroups-users >> > > > _______________________________________________ > javagroups-users mailing lis...@li...://lists.sourceforge.net/lists/listinfo/javagroups-users > > > -- > Bela Ban | http://www.jgroups.org > > _______________________________________________ > javagroups-users mailing list > jav...@li... > https://lists.sourceforge.net/lists/listinfo/javagroups-users > |
|
From: Questions/problems r. to u. J. <jav...@li...> - 2026-11-03 10:38:41
|
Hi Bobby I suspect that this is an old view change that was retransmitted until the members who were still in the view received it. If a member was *not* part of the view anymore, it would have rejected that view message. Again, this is only a suspicion. We could check this with a reproducer, but then again, I have other stuff on my todo list, and this seems to be an edge case. Shout if you disagree... :-) Cheers On 22.10.2026 13:07, Questions/problems related to using JGroups via javagroups-users wrote: > Hi again, > > Thanks for all of this. I have another ask about it: can you give a > brief explanation of why the view changes work out the way they do? > This is from the original email: > > --- start --- > The test case isolates node3 for 20s, reconnects it again and > waits 60s, then isolates it for 30s and reconnects it again. A few seconds > after that last reconnection, there is a view change for node3 and another > node (sometimes node1, sometimes node2), excluding the third node. That > third node doesn't see a view change until later when there is a merge of > all three again. Since the odd view change happens when all three > nodes are > available to each other, the customer would like to know why this is > happening. > --- end --- > > Listing out the view changes: > > 1) node3/node2 see a new view of just those two nodes. node1 does not > see a view change. (sometimes it's nodes 3/1 that see a change and 2 > does not) > 2) all three nodes see a merge view change containing all three. > > In other words, why does node3 and another node see a view change of > just the two of them? > > Thanks, > Bobby > > > On Thu, Oct 1, 2026 at 11:06 AM Questions/problems related to using > JGroups via javagroups-users <jav...@li...> > wrote: > > tcp_retries2=3 will likely work, but JGroups cannot handle the > underlying issue alone: if no packets are received at the (TCP/)IP > level, then all bets are off. > > For example, if no heartbeats are received, a member (or multiple > members) will get suspected and excluded from the cluster. > However, due to the same issue, the ensuing view will not get > delivered to all members until tcp_retries2 kicks in. > > This creates all sorts of issues that JGroups (and no other > software) cannot fix. > > On 01.10.2026 15:45, Questions/problems related to using JGroups > via javagroups-users wrote: >> On Thu, Oct 1, 2026 at 7:36 AM Questions/problems related to >> using JGroups via javagroups-users >> <jav...@li...> wrote: >> >> [...] >> 5.5.7? Super! Since this was released 3 weeks ago, is this >> already part of a product? >> >> >> Our next release will be in a couple months and will include >> 5.5.7. The customer had an earlier version, but I bumped up to >> the latest before looking at this test, reproduced the same thing >> there, etc etc. I just didn't want to make my email any longer >> than it already was. :) >> >> >> I'm wondering how the iptables test case relates to their >> actual problem. Could it be a firewall / switch issue? >> >> >> I don't think we'll know exactly what happened there, but this >> customer was interested enough to put in the time to come up with >> a way to reproduce the unexpected (to us) view changes. They had >> one of the three nodes on a different subnet, though there was no >> technical reason for it. I've suggested they put everything in >> one place, and hopefully they won't have this kind of issue again. >> >> Thank you for looking at this. I really just wanted to make sure >> that the TCP explanation wasn't a Claude hallucination that just >> happened to work out when I fiddled with the TCP settings. >> >> >> >> Your customer needs to evaluate this in terms of the tradeoff >> between quick detection of failures (tcp_retries2=3) and >> premature closing of a TCP connection (this takes longer with >> tcp_retries2=15). For example, Oracle RAC recommends a value >> of 3. >> >> The question is: is this an extreme edge case? Then I'd leave >> the tcp_retries2 value at 15, else reduce it. Setting >> tcp_retries2 affects the entire host, not just an individual >> process, so think about the consequences for other applications. >> >> >> I think this is just a case of "why does this happen?" I will >> tell them that they can reduce tcp_retries2 if it's important to >> them to handle this test case, but really they should leave it >> alone and get the nodes into a network that doesn't have this >> kind of intermittent disconnection. Even if JGroups can handle >> it, my product sitting on top still needs to communicate between >> the nodes fairly regularly. >> >> Cheers, and thanks as always for the quick help, >> Bobby >> >> >> >> >> _______________________________________________ >> javagroups-users mailing list >> jav...@li... >> https://lists.sourceforge.net/lists/listinfo/javagroups-users > > -- > Bela Ban |http://www.jgroups.org > > _______________________________________________ > javagroups-users mailing list > jav...@li... > https://lists.sourceforge.net/lists/listinfo/javagroups-users > > > > _______________________________________________ > javagroups-users mailing list > jav...@li... > https://lists.sourceforge.net/lists/listinfo/javagroups-users -- Bela Ban |http://www.jgroups.org |
|
From: Questions/problems r. to u. J. <jav...@li...> - 2026-10-22 11:07:52
|
Hi again, Thanks for all of this. I have another ask about it: can you give a brief explanation of why the view changes work out the way they do? This is from the original email: --- start --- The test case isolates node3 for 20s, reconnects it again and waits 60s, then isolates it for 30s and reconnects it again. A few seconds after that last reconnection, there is a view change for node3 and another node (sometimes node1, sometimes node2), excluding the third node. That third node doesn't see a view change until later when there is a merge of all three again. Since the odd view change happens when all three nodes are available to each other, the customer would like to know why this is happening. --- end --- Listing out the view changes: 1) node3/node2 see a new view of just those two nodes. node1 does not see a view change. (sometimes it's nodes 3/1 that see a change and 2 does not) 2) all three nodes see a merge view change containing all three. In other words, why does node3 and another node see a view change of just the two of them? Thanks, Bobby On Thu, Oct 1, 2026 at 11:06 AM Questions/problems related to using JGroups via javagroups-users <jav...@li...> wrote: > tcp_retries2=3 will likely work, but JGroups cannot handle the underlying > issue alone: if no packets are received at the (TCP/)IP level, then all > bets are off. > > For example, if no heartbeats are received, a member (or multiple members) > will get suspected and excluded from the cluster. However, due to the same > issue, the ensuing view will not get delivered to all members until > tcp_retries2 kicks in. > > This creates all sorts of issues that JGroups (and no other software) > cannot fix. > > On 01.10.2026 15:45, Questions/problems related to using JGroups via > javagroups-users wrote: > > On Thu, Oct 1, 2026 at 7:36 AM Questions/problems related to using JGroups > via javagroups-users <jav...@li...> wrote: > >> [...] >> 5.5.7? Super! Since this was released 3 weeks ago, is this already part >> of a product? >> > > Our next release will be in a couple months and will include 5.5.7. The > customer had an earlier version, but I bumped up to the latest before > looking at this test, reproduced the same thing there, etc etc. I just > didn't want to make my email any longer than it already was. :) > > >> >> I'm wondering how the iptables test case relates to their actual problem. >> Could it be a firewall / switch issue? >> > > I don't think we'll know exactly what happened there, but this customer > was interested enough to put in the time to come up with a way to reproduce > the unexpected (to us) view changes. They had one of the three nodes on a > different subnet, though there was no technical reason for it. I've > suggested they put everything in one place, and hopefully they won't have > this kind of issue again. > > Thank you for looking at this. I really just wanted to make sure that the > TCP explanation wasn't a Claude hallucination that just happened to work > out when I fiddled with the TCP settings. > > > >> Your customer needs to evaluate this in terms of the tradeoff between >> quick detection of failures (tcp_retries2=3) and premature closing of a TCP >> connection (this takes longer with tcp_retries2=15). For example, Oracle >> RAC recommends a value of 3. >> >> The question is: is this an extreme edge case? Then I'd leave the >> tcp_retries2 value at 15, else reduce it. Setting tcp_retries2 affects the >> entire host, not just an individual process, so think about the >> consequences for other applications. >> > > I think this is just a case of "why does this happen?" I will tell them > that they can reduce tcp_retries2 if it's important to them to handle this > test case, but really they should leave it alone and get the nodes into a > network that doesn't have this kind of intermittent disconnection. Even if > JGroups can handle it, my product sitting on top still needs to communicate > between the nodes fairly regularly. > > Cheers, and thanks as always for the quick help, > Bobby > > > > > _______________________________________________ > javagroups-users mailing lis...@li...://lists.sourceforge.net/lists/listinfo/javagroups-users > > > -- > Bela Ban | http://www.jgroups.org > > _______________________________________________ > javagroups-users mailing list > jav...@li... > https://lists.sourceforge.net/lists/listinfo/javagroups-users > |
|
From: Questions/problems r. to u. J. <jav...@li...> - 2026-10-01 14:52:52
|
tcp_retries2=3 will likely work, but JGroups cannot handle the underlying issue alone: if no packets are received at the (TCP/)IP level, then all bets are off. For example, if no heartbeats are received, a member (or multiple members) will get suspected and excluded from the cluster. However, due to the same issue, the ensuing view will not get delivered to all members until tcp_retries2 kicks in. This creates all sorts of issues that JGroups (and no other software) cannot fix. On 01.10.2026 15:45, Questions/problems related to using JGroups via javagroups-users wrote: > On Thu, Oct 1, 2026 at 7:36 AM Questions/problems related to using > JGroups via javagroups-users <jav...@li...> > wrote: > > [...] > 5.5.7? Super! Since this was released 3 weeks ago, is this already > part of a product? > > > Our next release will be in a couple months and will include > 5.5.7. The customer had an earlier version, but I bumped up to the > latest before looking at this test, reproduced the same thing there, > etc etc. I just didn't want to make my email any longer than it > already was. :) > > > I'm wondering how the iptables test case relates to their actual > problem. Could it be a firewall / switch issue? > > > I don't think we'll know exactly what happened there, but this > customer was interested enough to put in the time to come up with a > way to reproduce the unexpected (to us) view changes. They had one of > the three nodes on a different subnet, though there was no technical > reason for it. I've suggested they put everything in one place, and > hopefully they won't have this kind of issue again. > > Thank you for looking at this. I really just wanted to make sure that > the TCP explanation wasn't a Claude hallucination that just happened > to work out when I fiddled with the TCP settings. > > > > Your customer needs to evaluate this in terms of the tradeoff > between quick detection of failures (tcp_retries2=3) and premature > closing of a TCP connection (this takes longer with > tcp_retries2=15). For example, Oracle RAC recommends a value of 3. > > The question is: is this an extreme edge case? Then I'd leave the > tcp_retries2 value at 15, else reduce it. Setting tcp_retries2 > affects the entire host, not just an individual process, so think > about the consequences for other applications. > > > I think this is just a case of "why does this happen?" I will tell > them that they can reduce tcp_retries2 if it's important to them to > handle this test case, but really they should leave it alone and get > the nodes into a network that doesn't have this kind of intermittent > disconnection. Even if JGroups can handle it, my product sitting on > top still needs to communicate between the nodes fairly regularly. > > Cheers, and thanks as always for the quick help, > Bobby > > > > > _______________________________________________ > javagroups-users mailing list > jav...@li... > https://lists.sourceforge.net/lists/listinfo/javagroups-users -- Bela Ban |http://www.jgroups.org |
|
From: Questions/problems r. to u. J. <jav...@li...> - 2026-10-01 14:13:29
|
On Thu, Oct 1, 2026 at 7:36 AM Questions/problems related to using JGroups via javagroups-users <jav...@li...> wrote: > [...] > 5.5.7? Super! Since this was released 3 weeks ago, is this already part of > a product? > Our next release will be in a couple months and will include 5.5.7. The customer had an earlier version, but I bumped up to the latest before looking at this test, reproduced the same thing there, etc etc. I just didn't want to make my email any longer than it already was. :) > > I'm wondering how the iptables test case relates to their actual problem. > Could it be a firewall / switch issue? > I don't think we'll know exactly what happened there, but this customer was interested enough to put in the time to come up with a way to reproduce the unexpected (to us) view changes. They had one of the three nodes on a different subnet, though there was no technical reason for it. I've suggested they put everything in one place, and hopefully they won't have this kind of issue again. Thank you for looking at this. I really just wanted to make sure that the TCP explanation wasn't a Claude hallucination that just happened to work out when I fiddled with the TCP settings. > Your customer needs to evaluate this in terms of the tradeoff between > quick detection of failures (tcp_retries2=3) and premature closing of a TCP > connection (this takes longer with tcp_retries2=15). For example, Oracle > RAC recommends a value of 3. > > The question is: is this an extreme edge case? Then I'd leave the > tcp_retries2 value at 15, else reduce it. Setting tcp_retries2 affects the > entire host, not just an individual process, so think about the > consequences for other applications. > I think this is just a case of "why does this happen?" I will tell them that they can reduce tcp_retries2 if it's important to them to handle this test case, but really they should leave it alone and get the nodes into a network that doesn't have this kind of intermittent disconnection. Even if JGroups can handle it, my product sitting on top still needs to communicate between the nodes fairly regularly. Cheers, and thanks as always for the quick help, Bobby |
|
From: Questions/problems r. to u. J. <jav...@li...> - 2026-10-01 11:36:22
|
Hi Bobby On 28.09.2026 20:06, Questions/problems related to using JGroups via javagroups-users wrote: > Hi, > > A customer recently had some network issues that caused irregular view > changes in a three node cluster, e.g. two nodes seeing a view change > excluding the third, but the third did not see a change. They came up > with a little test case that shows this, and I wanted to check on an > explanation for it before getting back to them. > > I'll summarize here and add details down below. I'm using JGroups > v5.5.7, with a TCP based stack, 5.5.7? Super! Since this was released 3 weeks ago, is this already part of a product? > and the cluster has three nodes: node1, node2, and node3. I have the > FD_ALL timeout set to 50 seconds. node3 is the coordinator. The test > case isolates node3 for 20s, reconnects it again and waits 60s, then > isolates it for 30s and reconnects it again. A few seconds after that > last reconnection, there is a view change for node3 and another node > (sometimes node1, sometimes node2), excluding the third node. That > third node doesn't see a view change until later when there is a merge > of all three again. Since the odd view change happens when all three > nodes are available to each other, the customer would like to know why > this is happening. I'm wondering how the iptables test case relates to their actual problem. Could it be a firewall / switch issue? > I've dug into it with a lot of help from Claude, and what it's come up > with after lots of tests with TRACE level logging (the same behavior > happens with TRACE or INFO) is that the issue is down at the TCP > network layer. "During the outage, TCP backs off its retries to longer > and longer intervals, so after the network comes back a node's > messages (heartbeats included) can stay stuck in the connection until > the next retry, making the node look silent for much longer than the > outage itself." If I change tcp_retries2 from my default 15 down to 5, > I do see this test case pass. But I'm a complete newbie to this layer > of networking, and wanted to ask if the explanation makes sense or is > a lucky hallucination. Again, from Claude who was watching the logs: > > --- start --- > The key facts from the run with tcp_retries2=5: > - The stalled connections were aborted 15–20s into the 30s outage. > - New connections came up about 1s after the network returned. > - node3 suspected no one. > > With the default 15, the same test fails: node2 is suspected 50s > into the outage, 19s after the network was already back. > --- end --- > > My questions: > > - Is that explanation about TCP correct? I know almost nothing about > networking down at that level. > - Is there anything we can do to improve this? Cranking > up VERIFY_SUSPECT.timeout to 15 seconds avoids the view changes, but > so does adding that same amount of extra time to FD_ALL's timeout. > Personally, I don't care that much about handling any specific network > outage scenario, as other kinds of outages will still cause unexpected > behavior. I am fine with it as long as the cluster merges again later > and becomes healthy. (As it does in all of my tests.) Your customer needs to evaluate this in terms of the tradeoff between quick detection of failures (tcp_retries2=3) and premature closing of a TCP connection (this takes longer with tcp_retries2=15). For example, Oracle RAC recommends a value of 3. The question is: is this an extreme edge case? Then I'd leave the tcp_retries2 value at 15, else reduce it. Setting tcp_retries2 affects the entire host, not just an individual process, so think about the consequences for other applications. Closing TCP connections prematurely is not a problem for JGroups, as new connections will be established when needed, thus creating new TCP retransmission timers. So - from a JGroups perspective only - I'd recommend setting tcp_retries2 to a low value (3 or 5). Note that you could also experiment with TCP.conn_expire_time and TCP.reaper_interval and (possibly) TCP.enable_suspect_events: if conn_expire_time is exceeded, an 'idle' connection is closed by JGroups. Although, on second thought, this won't happen as messages sent over a connection will reset the last timestamp. Cheers, [1] https://access.redhat.com/solutions/726753 > Bela Ban | http://www.jgroups.org |
|
From: Questions/problems r. to u. J. <jav...@li...> - 2026-09-28 18:34:49
|
Hi,
A customer recently had some network issues that caused irregular view
changes in a three node cluster, e.g. two nodes seeing a view change
excluding the third, but the third did not see a change. They came up with
a little test case that shows this, and I wanted to check on an explanation
for it before getting back to them.
I'll summarize here and add details down below. I'm using JGroups v5.5.7,
with a TCP based stack, and the cluster has three nodes: node1, node2, and
node3. I have the FD_ALL timeout set to 50 seconds. node3 is the
coordinator. The test case isolates node3 for 20s, reconnects it again and
waits 60s, then isolates it for 30s and reconnects it again. A few seconds
after that last reconnection, there is a view change for node3 and another
node (sometimes node1, sometimes node2), excluding the third node. That
third node doesn't see a view change until later when there is a merge of
all three again. Since the odd view change happens when all three nodes are
available to each other, the customer would like to know why this is
happening.
I've dug into it with a lot of help from Claude, and what it's come up with
after lots of tests with TRACE level logging (the same behavior happens
with TRACE or INFO) is that the issue is down at the TCP network layer. "During
the outage, TCP backs off its retries to longer and longer intervals, so
after the network comes back a node's messages (heartbeats included) can
stay stuck in the connection until the next retry, making the node look
silent for much longer than the outage itself." If I change tcp_retries2
from my default 15 down to 5, I do see this test case pass. But I'm a
complete newbie to this layer of networking, and wanted to ask if the
explanation makes sense or is a lucky hallucination. Again, from Claude who
was watching the logs:
--- start ---
The key facts from the run with tcp_retries2=5:
- The stalled connections were aborted 15–20s into the 30s outage.
- New connections came up about 1s after the network returned.
- node3 suspected no one.
With the default 15, the same test fails: node2 is suspected 50s into the
outage, 19s after the network was already back.
--- end ---
My questions:
- Is that explanation about TCP correct? I know almost nothing about
networking down at that level.
- Is there anything we can do to improve this? Cranking
up VERIFY_SUSPECT.timeout to 15 seconds avoids the view changes, but so
does adding that same amount of extra time to FD_ALL's timeout. Personally,
I don't care that much about handling any specific network outage scenario,
as other kinds of outages will still cause unexpected behavior. I am fine
with it as long as the cluster merges again later and becomes healthy. (As
it does in all of my tests.)
I do get "failed sending graceful close to 172.17.0.3:7800: Broken pipe"
errors with tcp_retries2=5, but I only have that setting for testing. I was
not planning on telling customers to make that change.
------------------------------
Details for above here:
The test case is here, which is run on node3. Sometimes I have to run it
twice in a row to cause the view changes, but not always. I get the same
results whether using iptables or using docker network disconnect/connect.
--- start ---
echo closing
sudo iptables -A INPUT -s node1 -j DROP
sudo iptables -A OUTPUT -d node1 -j DROP
sudo iptables -A INPUT -s node2 -j DROP
sudo iptables -A OUTPUT -d node2 -j DROP
sleep 20
echo opening
sudo iptables -D INPUT -s node1 -j DROP
sudo iptables -D OUTPUT -d node1 -j DROP
sudo iptables -D INPUT -s node2 -j DROP
sudo iptables -D OUTPUT -d node2 -j DROP
sleep 60
echo closing
sudo iptables -A INPUT -s node1 -j DROP
sudo iptables -A OUTPUT -d node1 -j DROP
sudo iptables -A INPUT -s node2 -j DROP
sudo iptables -A OUTPUT -d node2 -j DROP
sleep 30
echo opening
sudo iptables -D INPUT -s node1 -j DROP
sudo iptables -D OUTPUT -d node1 -j DROP
sudo iptables -D INPUT -s node2 -j DROP
sudo iptables -D OUTPUT -d node2 -j DROP
sleep 60
echo "done sleep 60"
--- end ---
My stack, created programmatically, is here. I inherited this stack over a
decade ago, so there are probably improvements I could make that I don't
know about:
List<Protocol> stack = new ArrayList<>();
final Protocol tcp = new TCP()
.setValue("bind_addr", addr) // is user configurable
.setValue("bind_port", bindingPort) // is user configurable
.setValue("send_buf_size", 640000)
.setValue("sock_conn_timeout", 300)
.setValue("recv_buf_size", 5000000);
stack.add(tcp);
stack.add(new TCPPING()
.setValue("initial_hosts", list) // is user configurable
.setValue("send_cache_on_join", true)
.setValue("port_range", 0));
stack.add(new MERGE3()
.setValue("min_interval", 10000)
.setValue("max_interval", 30000));
FD_ALL fdAll = new FD_ALL();
fdAll.setValue("timeout", 50000); // is user configurable
// code ommitted that lowers 'interval' if timeout isn't at least
3*interval
stack.add(fdAll);
stack.add(new VERIFY_SUSPECT().setValue("timeout", 2000); // is
user configurable
stack.add(new BARRIER());
stack.add(new NAKACK2()
.setValue("use_mcast_xmit", false));
stack.add(new UNICAST3());
stack.add(new STABLE()
.setValue("desired_avg_gossip", 50000)
.setValue("max_bytes", 4000000));
stack.add(createAuthProtocol(efmNode));
stack.add(new GMS()
.setValue("join_timeout", getJGroupsJoinTimeout()));
stack.add(new MFC()
.setValue("max_credits", 2000000)
.setValue("min_credits", 800000));
stack.add(new FRAG2());
stack.add(new STATE_TRANSFER());
JChannel retVal = new JChannel(stack);
I can send logs from the three nodes if that helps anything.
Thank you,
Bobby Bissett
|
|
From: Questions/problems r. to u. J. <jav...@li...> - 2023-09-12 11:09:27
|
FYI, I just released 5.3. Details here: http://belaban.blogspot.com/2023/09/jgroups-53-released.html Cheers -- Bela Ban | http://www.jgroups.org |
|
From: Questions/problems r. to u. J. <jav...@li...> - 2022-11-22 01:29:38
|
Thanks for the confirmation Bela, I was easily able to modify the
Infinispan configuration to include the FORK [1] and the application is no
longer prone to random OOMs during bootstrap.
Cheers,
Johnathan
[1]
<config xmlns="urn:org:jgroups"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:org:jgroups
http://www.jgroups.org/schema/jgroups-4.0.xsd">
...
<FORK>
<fork-stacks>
<fork-stack id="hijack-stack"/>
</fork-stacks>
</FORK>
<FRAG3 />
</config>
if (cacheManager.getTransport() instanceof JGroupsTransport) {
JGroupsTransport jGroupsTransport = (JGroupsTransport)
cacheManager.getTransport();
ProtocolStack stack = jGroupsTransport.getChannel().getProtocolStack();
Class<? extends Protocol> neighborProtocol =
stack.findProtocol(FRAG2.class) != null ?
FRAG2.class : FRAG3.class;
channel = new ForkChannel(jGroupsTransport.getChannel(),
"hijack-stack",
"lead-hijacker",
false,
ProtocolStack.Position.ABOVE,
neighborProtocol);
On Fri, 21 Oct 2022, 16:40 Questions/problems related to using JGroups via
javagroups-users, <jav...@li...> wrote:
> Hi Jonathan
>
> yes, the best solution is to define FORK in the JGroups section of the
> Infinispan configuration, as you mentioned.
>
> If you don't control the configuration, then it gets a bit more
> tricky... you could (*before sending any brodcast traffic*) insert FORK
> dynamically into every JChannel instance.
> To do that, you need to get the JChannel; IIRC, via (paraphrased)
> cache.getExtendedCache().getRpcManager().getTransport(), downcast it to
> JGroupsTransport, then call getChannel().
>
> Once you have JChannel, call
> channel.getProtocolStack().insertProtocolAtTop(new FORK(...));
> Hope this helps,
>
> [1] http://www.jgroups.org/manual5/index.html#ForkChannel
>
> On 21.10.22 16:02, Questions/problems related to using JGroups wrote:
> > Hi all,
> >
> > We have run into an interesting race condition when attempting to use a
> > fork channel in our application, we more or less follow what Bela wrote
> > here
> >
> http://belaban.blogspot.com/2013/08/how-to-hijack-jgroups-channel-inside.html
> <
> http://belaban.blogspot.com/2013/08/how-to-hijack-jgroups-channel-inside.html>
> .
> >
> > When the cluster comes up and receives views the node broadcasts
> > information about itself over the fork channel so the cluster knows what
> > each node is capable of handling.
> >
> > Unfortunately from what I can see there is a race condition on bootstrap
> > where the JGroups stack is started and receives a fork message before
> > the fork is inserted into the stack (fork not present in stack trace)
> > [1] which results in garbage / unknown data passing through the
> > Infinispan marshaller.. if you are lucky enough it will read an
> > extremely large int and try and allocate that into a byte array
> > resulting in the JVM to throw an OOM or NegativeArraySizeException
> >
> > I believe one possible solution is to define the fork inside the
> > jgroups.xml which is used to create the initial jgroups stack which
> > would hopefully discard fork channel messages (until the message
> > listener is registered) and not pass them up the stack resulting in
> > undefined behaviour.
> >
> > I had a look at the fork documentation but there are not many examples,
> > does my possible solution seem feasible or does someone have alternative
> > solutions? I am currently looking through the Infinispan code to see if
> > there is any way to decorate jgroups before it starts.
> >
> > Thanks in advance,
> >
> > Johnathan
> >
> > [1]
> >
> > 2022-10-20 10:42:28,515 ERROR [jgroups-89,service-2]
> > (org.infinispan.CLUSTER) ISPN000474: Error processing request 0@service-2
> > java.lang.NegativeArraySizeException: -436207616
> > at
> >
> org.infinispan.marshall.core.GlobalMarshaller.readUnknown(GlobalMarshaller.java:904)
> ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
> > at
> >
> org.infinispan.marshall.core.GlobalMarshaller.readUnknown(GlobalMarshaller.java:891)
> ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
> > at
> >
> org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:715)
> ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
> > at
> >
> org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:358)
> ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
> > at
> >
> org.infinispan.marshall.core.GlobalMarshaller.objectFromObjectInput(GlobalMarshaller.java:192)
> ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
> > at
> >
> org.infinispan.marshall.core.GlobalMarshaller.objectFromByteBuffer(GlobalMarshaller.java:221)
> ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
> > at
> >
> org.infinispan.remoting.transport.jgroups.JGroupsTransport.processRequest(JGroupsTransport.java:1361)
> ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
> > at
> >
> org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1301)
> ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
> > at
> >
> org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$300(JGroupsTransport.java:130)
> ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
> > at
> >
> org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.lambda$up$0(JGroupsTransport.java:1450)
> ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
> > at org.jgroups.util.MessageBatch.forEach(MessageBatch.java:318)
> > ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at
> >
> org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1450)
> ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final]
> > at org.jgroups.JChannel.up(JChannel.java:796)
> > ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:903)
> > ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at org.jgroups.protocols.FRAG3.up(FRAG3.java:187)
> > ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
> > ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at org.jgroups.stack.Protocol.up(Protocol.java:338)
> > ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:297)
> > ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at org.jgroups.protocols.UNICAST3.deliverBatch(UNICAST3.java:1071)
> > ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at org.jgroups.protocols.UNICAST3.removeAndDeliver(UNICAST3.java:886)
> > ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at org.jgroups.protocols.UNICAST3.handleBatchReceived(UNICAST3.java:852)
> > ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:501)
> > ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:689)
> > ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at org.jgroups.stack.Protocol.up(Protocol.java:338)
> > ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at org.jgroups.protocols.FailureDetection.up(FailureDetection.java:197)
> > ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at org.jgroups.stack.Protocol.up(Protocol.java:338)
> > ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at org.jgroups.stack.Protocol.up(Protocol.java:338)
> > ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at org.jgroups.stack.Protocol.up(Protocol.java:338)
> > ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at org.jgroups.protocols.TP.passBatchUp(TP.java:1408)
> > ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at
> >
> org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.passBatchUp(MaxOneThreadPerSender.java:284)
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at
> >
> org.jgroups.util.SubmitToThreadPool$BatchHandler.run(SubmitToThreadPool.java:136)
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at
> >
> org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.run(MaxOneThreadPerSender.java:273)
> ~[jgroups-4.2.1.Final.jar:4.2.1.Final]
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> ~[?:?]
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> ~[?:?]
> > at java.lang.Thread.run(Thread.java:829) ~[?:?]
> >
> >
> > _______________________________________________
> > javagroups-users mailing list
> > jav...@li...
> > https://lists.sourceforge.net/lists/listinfo/javagroups-users
>
> --
> Bela Ban | http://www.jgroups.org
>
>
>
> _______________________________________________
> javagroups-users mailing list
> jav...@li...
> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>
|
|
From: Questions/problems r. to u. J. <jav...@li...> - 2022-10-21 14:39:08
|
Hi Jonathan yes, the best solution is to define FORK in the JGroups section of the Infinispan configuration, as you mentioned. If you don't control the configuration, then it gets a bit more tricky... you could (*before sending any brodcast traffic*) insert FORK dynamically into every JChannel instance. To do that, you need to get the JChannel; IIRC, via (paraphrased) cache.getExtendedCache().getRpcManager().getTransport(), downcast it to JGroupsTransport, then call getChannel(). Once you have JChannel, call channel.getProtocolStack().insertProtocolAtTop(new FORK(...)); Hope this helps, [1] http://www.jgroups.org/manual5/index.html#ForkChannel On 21.10.22 16:02, Questions/problems related to using JGroups wrote: > Hi all, > > We have run into an interesting race condition when attempting to use a > fork channel in our application, we more or less follow what Bela wrote > here > http://belaban.blogspot.com/2013/08/how-to-hijack-jgroups-channel-inside.html <http://belaban.blogspot.com/2013/08/how-to-hijack-jgroups-channel-inside.html> . > > When the cluster comes up and receives views the node broadcasts > information about itself over the fork channel so the cluster knows what > each node is capable of handling. > > Unfortunately from what I can see there is a race condition on bootstrap > where the JGroups stack is started and receives a fork message before > the fork is inserted into the stack (fork not present in stack trace) > [1] which results in garbage / unknown data passing through the > Infinispan marshaller.. if you are lucky enough it will read an > extremely large int and try and allocate that into a byte array > resulting in the JVM to throw an OOM or NegativeArraySizeException > > I believe one possible solution is to define the fork inside the > jgroups.xml which is used to create the initial jgroups stack which > would hopefully discard fork channel messages (until the message > listener is registered) and not pass them up the stack resulting in > undefined behaviour. > > I had a look at the fork documentation but there are not many examples, > does my possible solution seem feasible or does someone have alternative > solutions? I am currently looking through the Infinispan code to see if > there is any way to decorate jgroups before it starts. > > Thanks in advance, > > Johnathan > > [1] > > 2022-10-20 10:42:28,515 ERROR [jgroups-89,service-2] > (org.infinispan.CLUSTER) ISPN000474: Error processing request 0@service-2 > java.lang.NegativeArraySizeException: -436207616 > at > org.infinispan.marshall.core.GlobalMarshaller.readUnknown(GlobalMarshaller.java:904) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final] > at > org.infinispan.marshall.core.GlobalMarshaller.readUnknown(GlobalMarshaller.java:891) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final] > at > org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:715) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final] > at > org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:358) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final] > at > org.infinispan.marshall.core.GlobalMarshaller.objectFromObjectInput(GlobalMarshaller.java:192) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final] > at > org.infinispan.marshall.core.GlobalMarshaller.objectFromByteBuffer(GlobalMarshaller.java:221) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final] > at > org.infinispan.remoting.transport.jgroups.JGroupsTransport.processRequest(JGroupsTransport.java:1361) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final] > at > org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1301) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final] > at > org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$300(JGroupsTransport.java:130) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final] > at > org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.lambda$up$0(JGroupsTransport.java:1450) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final] > at org.jgroups.util.MessageBatch.forEach(MessageBatch.java:318) > ~[jgroups-4.2.1.Final.jar:4.2.1.Final] > at > org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1450) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final] > at org.jgroups.JChannel.up(JChannel.java:796) > ~[jgroups-4.2.1.Final.jar:4.2.1.Final] > at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:903) > ~[jgroups-4.2.1.Final.jar:4.2.1.Final] > at org.jgroups.protocols.FRAG3.up(FRAG3.java:187) > ~[jgroups-4.2.1.Final.jar:4.2.1.Final] > at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) > ~[jgroups-4.2.1.Final.jar:4.2.1.Final] > at org.jgroups.stack.Protocol.up(Protocol.java:338) > ~[jgroups-4.2.1.Final.jar:4.2.1.Final] > at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:297) > ~[jgroups-4.2.1.Final.jar:4.2.1.Final] > at org.jgroups.protocols.UNICAST3.deliverBatch(UNICAST3.java:1071) > ~[jgroups-4.2.1.Final.jar:4.2.1.Final] > at org.jgroups.protocols.UNICAST3.removeAndDeliver(UNICAST3.java:886) > ~[jgroups-4.2.1.Final.jar:4.2.1.Final] > at org.jgroups.protocols.UNICAST3.handleBatchReceived(UNICAST3.java:852) > ~[jgroups-4.2.1.Final.jar:4.2.1.Final] > at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:501) > ~[jgroups-4.2.1.Final.jar:4.2.1.Final] > at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:689) > ~[jgroups-4.2.1.Final.jar:4.2.1.Final] > at org.jgroups.stack.Protocol.up(Protocol.java:338) > ~[jgroups-4.2.1.Final.jar:4.2.1.Final] > at org.jgroups.protocols.FailureDetection.up(FailureDetection.java:197) > ~[jgroups-4.2.1.Final.jar:4.2.1.Final] > at org.jgroups.stack.Protocol.up(Protocol.java:338) > ~[jgroups-4.2.1.Final.jar:4.2.1.Final] > at org.jgroups.stack.Protocol.up(Protocol.java:338) > ~[jgroups-4.2.1.Final.jar:4.2.1.Final] > at org.jgroups.stack.Protocol.up(Protocol.java:338) > ~[jgroups-4.2.1.Final.jar:4.2.1.Final] > at org.jgroups.protocols.TP.passBatchUp(TP.java:1408) > ~[jgroups-4.2.1.Final.jar:4.2.1.Final] > at > org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.passBatchUp(MaxOneThreadPerSender.java:284) ~[jgroups-4.2.1.Final.jar:4.2.1.Final] > at > org.jgroups.util.SubmitToThreadPool$BatchHandler.run(SubmitToThreadPool.java:136) ~[jgroups-4.2.1.Final.jar:4.2.1.Final] > at > org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.run(MaxOneThreadPerSender.java:273) ~[jgroups-4.2.1.Final.jar:4.2.1.Final] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] > at java.lang.Thread.run(Thread.java:829) ~[?:?] > > > _______________________________________________ > javagroups-users mailing list > jav...@li... > https://lists.sourceforge.net/lists/listinfo/javagroups-users -- Bela Ban | http://www.jgroups.org |
|
From: Questions/problems r. to u. J. <jav...@li...> - 2022-10-21 14:02:42
|
Hi all, We have run into an interesting race condition when attempting to use a fork channel in our application, we more or less follow what Bela wrote here http://belaban.blogspot.com/2013/08/how-to-hijack-jgroups-channel-inside.html . When the cluster comes up and receives views the node broadcasts information about itself over the fork channel so the cluster knows what each node is capable of handling. Unfortunately from what I can see there is a race condition on bootstrap where the JGroups stack is started and receives a fork message before the fork is inserted into the stack (fork not present in stack trace) [1] which results in garbage / unknown data passing through the Infinispan marshaller.. if you are lucky enough it will read an extremely large int and try and allocate that into a byte array resulting in the JVM to throw an OOM or NegativeArraySizeException I believe one possible solution is to define the fork inside the jgroups.xml which is used to create the initial jgroups stack which would hopefully discard fork channel messages (until the message listener is registered) and not pass them up the stack resulting in undefined behaviour. I had a look at the fork documentation but there are not many examples, does my possible solution seem feasible or does someone have alternative solutions? I am currently looking through the Infinispan code to see if there is any way to decorate jgroups before it starts. Thanks in advance, Johnathan [1] 2022-10-20 10:42:28,515 ERROR [jgroups-89,service-2] (org.infinispan.CLUSTER) ISPN000474: Error processing request 0@service-2 java.lang.NegativeArraySizeException: -436207616 at org.infinispan.marshall.core.GlobalMarshaller.readUnknown(GlobalMarshaller.java:904) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final] at org.infinispan.marshall.core.GlobalMarshaller.readUnknown(GlobalMarshaller.java:891) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final] at org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:715) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final] at org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:358) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final] at org.infinispan.marshall.core.GlobalMarshaller.objectFromObjectInput(GlobalMarshaller.java:192) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final] at org.infinispan.marshall.core.GlobalMarshaller.objectFromByteBuffer(GlobalMarshaller.java:221) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processRequest(JGroupsTransport.java:1361) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1301) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$300(JGroupsTransport.java:130) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final] at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.lambda$up$0(JGroupsTransport.java:1450) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final] at org.jgroups.util.MessageBatch.forEach(MessageBatch.java:318) ~[jgroups-4.2.1.Final.jar:4.2.1.Final] at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1450) ~[infinispan-core-11.0.1.Final.jar:11.0.1.Final] at org.jgroups.JChannel.up(JChannel.java:796) ~[jgroups-4.2.1.Final.jar:4.2.1.Final] at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:903) ~[jgroups-4.2.1.Final.jar:4.2.1.Final] at org.jgroups.protocols.FRAG3.up(FRAG3.java:187) ~[jgroups-4.2.1.Final.jar:4.2.1.Final] at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) ~[jgroups-4.2.1.Final.jar:4.2.1.Final] at org.jgroups.stack.Protocol.up(Protocol.java:338) ~[jgroups-4.2.1.Final.jar:4.2.1.Final] at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:297) ~[jgroups-4.2.1.Final.jar:4.2.1.Final] at org.jgroups.protocols.UNICAST3.deliverBatch(UNICAST3.java:1071) ~[jgroups-4.2.1.Final.jar:4.2.1.Final] at org.jgroups.protocols.UNICAST3.removeAndDeliver(UNICAST3.java:886) ~[jgroups-4.2.1.Final.jar:4.2.1.Final] at org.jgroups.protocols.UNICAST3.handleBatchReceived(UNICAST3.java:852) ~[jgroups-4.2.1.Final.jar:4.2.1.Final] at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:501) ~[jgroups-4.2.1.Final.jar:4.2.1.Final] at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:689) ~[jgroups-4.2.1.Final.jar:4.2.1.Final] at org.jgroups.stack.Protocol.up(Protocol.java:338) ~[jgroups-4.2.1.Final.jar:4.2.1.Final] at org.jgroups.protocols.FailureDetection.up(FailureDetection.java:197) ~[jgroups-4.2.1.Final.jar:4.2.1.Final] at org.jgroups.stack.Protocol.up(Protocol.java:338) ~[jgroups-4.2.1.Final.jar:4.2.1.Final] at org.jgroups.stack.Protocol.up(Protocol.java:338) ~[jgroups-4.2.1.Final.jar:4.2.1.Final] at org.jgroups.stack.Protocol.up(Protocol.java:338) ~[jgroups-4.2.1.Final.jar:4.2.1.Final] at org.jgroups.protocols.TP.passBatchUp(TP.java:1408) ~[jgroups-4.2.1.Final.jar:4.2.1.Final] at org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.passBatchUp(MaxOneThreadPerSender.java:284) ~[jgroups-4.2.1.Final.jar:4.2.1.Final] at org.jgroups.util.SubmitToThreadPool$BatchHandler.run(SubmitToThreadPool.java:136) ~[jgroups-4.2.1.Final.jar:4.2.1.Final] at org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.run(MaxOneThreadPerSender.java:273) ~[jgroups-4.2.1.Final.jar:4.2.1.Final] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:829) ~[?:?] |
|
From: Questions/problems r. to u. J. <jav...@li...> - 2022-10-17 14:06:13
|
On 17.10.22 14:30, Questions/problems related to using JGroups wrote: > Hi, > > I see that the suspect() method is only called on the coordinator node > in my cluster when another node disappears; no other nodes in the > cluster get the message. Is this the expected behavior? The docs don't > say one way or the other: > > http://www.jgroups.org/javadoc4/org/jgroups/MembershipListener.html#suspect(org.jgroups.Address) <http://www.jgroups.org/javadoc4/org/jgroups/MembershipListener.html#suspect(org.jgroups.Address)> Yes, this is correct: coordinator (or newly promoted coords) get this callback only. > We're using jgroups 4.1.8 currently. Note that the suspect() / unsuspect() were removed in 5.0. We wrote code, way back with > jgroups 3.4, where every node needs to know if a member left the view > suspect or not. IMO there's better ways of doing this, as suspect() is only an indication and may never result in a view change. If a member P needs to leave gracefully, I'd have P broadcast a LEAVING_GRACEFULLY message to all, before it leaves. Everyone receives this message and caches it. When the actual view change arrives, members part of the previous view but not the current view, and *not* part of the cache, crashed. All others left gracefully. The cache needs to be adjusted on every view change. > The nodes use this to compare the current cluster size > to a "stable" size that changes more slowly (changes to match cluster > size right away if nodes leave gracefully since that means a human > caused it on purpose). So either suspect() was called on every node back > then or we really missed something during testing. It was seven years > ago so we could have definitely missed it. > > Can send more information if you'd like it, but is this the way it's > supposed to work? Is there anything else I can do have suspect() called > on each node? > > (If not, I currently think my best option is to have non-coordinator > nodes treat any decrease in view size as suspect until, after the view > change, the coordinator can send them a message telling them that a node > left gracefully and that the current actual size is "stable.") > > Thank you, > Bobby > > > > _______________________________________________ > javagroups-users mailing list > jav...@li... > https://lists.sourceforge.net/lists/listinfo/javagroups-users -- Bela Ban | http://www.jgroups.org |
|
From: Questions/problems r. to u. J. <jav...@li...> - 2022-10-17 12:58:03
|
Hi, I see that the suspect() method is only called on the coordinator node in my cluster when another node disappears; no other nodes in the cluster get the message. Is this the expected behavior? The docs don't say one way or the other: http://www.jgroups.org/javadoc4/org/jgroups/MembershipListener.html#suspect(org.jgroups.Address) We're using jgroups 4.1.8 currently. We wrote code, way back with jgroups 3.4, where every node needs to know if a member left the view suspect or not. The nodes use this to compare the current cluster size to a "stable" size that changes more slowly (changes to match cluster size right away if nodes leave gracefully since that means a human caused it on purpose). So either suspect() was called on every node back then or we really missed something during testing. It was seven years ago so we could have definitely missed it. Can send more information if you'd like it, but is this the way it's supposed to work? Is there anything else I can do have suspect() called on each node? (If not, I currently think my best option is to have non-coordinator nodes treat any decrease in view size as suspect until, after the view change, the coordinator can send them a message telling them that a node left gracefully and that the current actual size is "stable.") Thank you, Bobby |
|
From: Questions/problems r. to u. J. <jav...@li...> - 2022-06-10 10:25:09
|
On 06.06.22 22:15, Questions/problems related to using JGroups wrote: > Although, looking at this again, I think we might not be talking about > the same setup. From this: > > > > On Wed, May 26, 2021 at 3:26 AM Questions/problems related to using > JGroups via javagroups-users <jav...@li... > <mailto:jav...@li...>> wrote: > > [....] > > > > > Right, and what they want is some way to fully remove a node > from a > > cluster. I.e. the cluster stops trying to contact that address. > > > Then you would have to remove the 130 node from the old cluster's > initial_hosts (TCPPING) and TCP's logical address cache. Either by > restarting, or by programmatically removing it. This can get > complex > quickly though, as you'd have to maintain a list of ports per > cluster. > > > Each cluster is separate from all the others, so I don't know what I > would need to keep in this list or why a cluster would need it. Referring to my previous email: if you use FILE_PING, each cluster has a _separate_ directory (the cluster name) under which the discovery info is stored. > If a cluster has A/B/C/D in it, and the code sees that D leaves the cluster > without going suspect first, can I programmatically do these? For TCP, it's complicated, but doable. Among other things you'd have to: - Close all TCP connections to D - Close all connections to D in UNICAST3, too - Remove D's info from the address cache (contents: 'probe.sh uuids') - Remove D's information from all instances of TCPPING (initial_hosts and dynamic_hosts) Again, using a dynamic discovery protocol such as FILE_PING makes more sense here. > - set new initial_hosts on the existing TCPPING protocol in my stack to > include only A/B/C > - access the logical address cache and remove the address Yes, but this is not enough (see above). > I mean, I know I can hack the TCPPING again, but didn't know that would > have any effect on the existing channel and members. I don't know > offhand how to access the address cache, which I think is all I'm > missing to experiment with this. Pseudo code: TP tp=channel.getProtocolStack().getTransport(); LazyRemovalCache cache=tp.getLogicalAddressCache(); cache.remove(address, true); // force removal > If I can do the above then I think that > solves the issue -- if a suspect member leaves the view I won't do > anything, because we want to keep trying it in case it was disconnected > and reconnected. But if a member leaves gracefully and the above is all > I need to make the cluster forget about it, that's great and means we > wouldn't have to change any startup features for the customers. > > Thanks again, > Bobby > > > > _______________________________________________ > javagroups-users mailing list > jav...@li... > https://lists.sourceforge.net/lists/listinfo/javagroups-users -- Bela Ban | http://www.jgroups.org |
|
From: Questions/problems r. to u. J. <jav...@li...> - 2022-06-10 09:47:52
|
Hi Bobby On 06.06.22 21:08, Questions/problems related to using JGroups wrote: > Hi again Bela et al, > > We've finally come back to this issue after not working on the product > for a while. I'm keeping the context all below, but the short version > was that we use TCPPING and, if someone removes a node with address X > and, later, starts a new cluster that includes the address, the old > cluster keeps trying to find its lost buddy at X. Right, and I suggested using a dynamic discovery protocol, *not* TCPPING. > We're still back on v4.1.8 and I wanted to ask if the suggestion below, > i.e. use TCPGOSSIP or FILE_PING (this is for in-house deployments on > their own networks) is the most appropriate, and if there would be any > benefit for this particular issue by moving to v5.X? There are loads of benefits by moving to 5.x :-) But, specifically to this case, only the ability to have multiple discovery protocols in the same stack would be beneficial here. I guess MULTI_PING in 4.x might do the same job though... > The way they run > things now is to put host:port info for each node in a file and then > start the applications, which read that file to set initial hosts. So > FILE_PING might be the best for them so that we don't need to have any > new processes running. Yes, the benefits/drawbacks of FILE_PING are + No additional process needed + All processes access a shared dir, e.g. on NFS - NFS adds overhead (but only for discovery) + The discovery info is human-readable, and can thus be modified manually (if needed) > Thanks, > Bobby > > On Wed, May 26, 2021 at 3:26 AM Questions/problems related to using > JGroups via javagroups-users <jav...@li... > <mailto:jav...@li...>> wrote: > > > > On 25.05.21 18:59, Questions/problems related to using JGroups wrote: > > On Tue, May 25, 2021 at 10:30 AM Questions/problems related to using > > JGroups via javagroups-users > <jav...@li... > <mailto:jav...@li...> > > <mailto:jav...@li... > <mailto:jav...@li...>>> wrote: > > > > Hi Bobby > > apologies for the delay! > > > > > > No problem -- thanks for looking. > > > > > > You cannot have the old cluster's initial_hosts be > 128,129,130 and the > > new one has the overlapping range 130,131. > > > > > > That's the problem. The customer has lots of nodes, clusters that > grow > > and shrink, and they're going to reuse the same IP addresses > eventually. > > > Then using TCPPING for the discovery is the wrong solution; it is > designed for a static cluster with a fixed and known membership. > > For the above requirements, I'd rather recommend: > * A dynamic discovery mechanism (TCPGOSSIP, FILE_PING, GOOGLE_PING etc) > * Emphemeral ports > * A new (different) cluster name for each new cluster that is started > > > > The old cluster will try to contact 130 (e.g. trying to > merge), thereby > > send its information to 130. > > > > > > Right, and what they want is some way to fully remove a node from a > > cluster. I.e. the cluster stops trying to contact that address. > > > Then you would have to remove the 130 node from the old cluster's > initial_hosts (TCPPING) and TCP's logical address cache. Either by > restarting, or by programmatically removing it. This can get complex > quickly though, as you'd have to maintain a list of ports per cluster. > > The first solution above is much better IMO. > > > > What is it you're trying to achieve? > > > > > > Simply to take a node out of a cluster when it's not needed, then > later > > reuse the address of that node with a different cluster. If I > change the > > cluster names (same port though) then I still get constant > warnings, like: > > JGRP000012: discarded message from different cluster <old> (our > cluster > > is <new>). Sender was <some addr> > > > > We can suggest that they restart the cluster after removing a > node, but > > I don't know if that will work for them. I'll also try using > different > > ports for different clusters and see how that works for them. > > That will certainly work, but - again - you'd have to maintain ports > numbers for each cluster. Registration service? Excel spreadsheet? > > > > Given the size of the company in question, I can see that it > might be hard to > > coordinate that and eventually they'll get back in the same > situation > > where a previously used address is being used again with the same > port > > it used the last time. > > Right. So I have to come back to my suggestion of not using TCPPING! > Cheers, > > > > Thanks, > > Bobby > > > > > > > > _______________________________________________ > > javagroups-users mailing list > > jav...@li... > <mailto:jav...@li...> > > https://lists.sourceforge.net/lists/listinfo/javagroups-users > <https://lists.sourceforge.net/lists/listinfo/javagroups-users> > > > > -- > Bela Ban | http://www.jgroups.org <http://www.jgroups.org> > > > > _______________________________________________ > javagroups-users mailing list > jav...@li... > <mailto:jav...@li...> > https://lists.sourceforge.net/lists/listinfo/javagroups-users > <https://lists.sourceforge.net/lists/listinfo/javagroups-users> > > > > _______________________________________________ > javagroups-users mailing list > jav...@li... > https://lists.sourceforge.net/lists/listinfo/javagroups-users -- Bela Ban | http://www.jgroups.org |
|
From: Questions/problems r. to u. J. <jav...@li...> - 2022-06-06 20:40:05
|
Although, looking at this again, I think we might not be talking about the same setup. From this: > > On Wed, May 26, 2021 at 3:26 AM Questions/problems related to using > JGroups via javagroups-users <jav...@li...> > wrote: > >> [....] >> >> > >> > Right, and what they want is some way to fully remove a node from a >> > cluster. I.e. the cluster stops trying to contact that address. >> >> >> Then you would have to remove the 130 node from the old cluster's >> initial_hosts (TCPPING) and TCP's logical address cache. Either by >> restarting, or by programmatically removing it. This can get complex >> quickly though, as you'd have to maintain a list of ports per cluster. > > Each cluster is separate from all the others, so I don't know what I would need to keep in this list or why a cluster would need it. If a cluster has A/B/C/D in it, and the code sees that D leaves the cluster without going suspect first, can I programmatically do these? - set new initial_hosts on the existing TCPPING protocol in my stack to include only A/B/C - access the logical address cache and remove the address I mean, I know I can hack the TCPPING again, but didn't know that would have any effect on the existing channel and members. I don't know offhand how to access the address cache, which I think is all I'm missing to experiment with this. If I can do the above then I think that solves the issue -- if a suspect member leaves the view I won't do anything, because we want to keep trying it in case it was disconnected and reconnected. But if a member leaves gracefully and the above is all I need to make the cluster forget about it, that's great and means we wouldn't have to change any startup features for the customers. Thanks again, Bobby |
|
From: Questions/problems r. to u. J. <jav...@li...> - 2022-06-06 19:35:18
|
Hi again Bela et al, We've finally come back to this issue after not working on the product for a while. I'm keeping the context all below, but the short version was that we use TCPPING and, if someone removes a node with address X and, later, starts a new cluster that includes the address, the old cluster keeps trying to find its lost buddy at X. We're still back on v4.1.8 and I wanted to ask if the suggestion below, i.e. use TCPGOSSIP or FILE_PING (this is for in-house deployments on their own networks) is the most appropriate, and if there would be any benefit for this particular issue by moving to v5.X? The way they run things now is to put host:port info for each node in a file and then start the applications, which read that file to set initial hosts. So FILE_PING might be the best for them so that we don't need to have any new processes running. Thanks, Bobby On Wed, May 26, 2021 at 3:26 AM Questions/problems related to using JGroups via javagroups-users <jav...@li...> wrote: > > > On 25.05.21 18:59, Questions/problems related to using JGroups wrote: > > On Tue, May 25, 2021 at 10:30 AM Questions/problems related to using > > JGroups via javagroups-users <jav...@li... > > <mailto:jav...@li...>> wrote: > > > > Hi Bobby > > apologies for the delay! > > > > > > No problem -- thanks for looking. > > > > > > You cannot have the old cluster's initial_hosts be 128,129,130 and > the > > new one has the overlapping range 130,131. > > > > > > That's the problem. The customer has lots of nodes, clusters that grow > > and shrink, and they're going to reuse the same IP addresses eventually. > > > Then using TCPPING for the discovery is the wrong solution; it is > designed for a static cluster with a fixed and known membership. > > For the above requirements, I'd rather recommend: > * A dynamic discovery mechanism (TCPGOSSIP, FILE_PING, GOOGLE_PING etc) > * Emphemeral ports > * A new (different) cluster name for each new cluster that is started > > > > The old cluster will try to contact 130 (e.g. trying to merge), > thereby > > send its information to 130. > > > > > > Right, and what they want is some way to fully remove a node from a > > cluster. I.e. the cluster stops trying to contact that address. > > > Then you would have to remove the 130 node from the old cluster's > initial_hosts (TCPPING) and TCP's logical address cache. Either by > restarting, or by programmatically removing it. This can get complex > quickly though, as you'd have to maintain a list of ports per cluster. > > The first solution above is much better IMO. > > > > What is it you're trying to achieve? > > > > > > Simply to take a node out of a cluster when it's not needed, then later > > reuse the address of that node with a different cluster. If I change the > > cluster names (same port though) then I still get constant warnings, > like: > > JGRP000012: discarded message from different cluster <old> (our cluster > > is <new>). Sender was <some addr> > > > > We can suggest that they restart the cluster after removing a node, but > > I don't know if that will work for them. I'll also try using different > > ports for different clusters and see how that works for them. > > That will certainly work, but - again - you'd have to maintain ports > numbers for each cluster. Registration service? Excel spreadsheet? > > > > Given the size of the company in question, I can see that it might be > hard to > > coordinate that and eventually they'll get back in the same situation > > where a previously used address is being used again with the same port > > it used the last time. > > Right. So I have to come back to my suggestion of not using TCPPING! > Cheers, > > > > Thanks, > > Bobby > > > > > > > > _______________________________________________ > > javagroups-users mailing list > > jav...@li... > > https://lists.sourceforge.net/lists/listinfo/javagroups-users > > > > -- > Bela Ban | http://www.jgroups.org > > > > _______________________________________________ > javagroups-users mailing list > jav...@li... > https://lists.sourceforge.net/lists/listinfo/javagroups-users > |
|
From: Questions/problems r. to u. J. <jav...@li...> - 2021-11-22 15:27:41
|
Hi again, Thanks for this -- seems like an obvious answer but I wanted to be sure. On Mon, Nov 22, 2021 at 3:43 AM Questions/problems related to using JGroups via javagroups-users <jav...@li...> wrote: > > If they use this as a kind of health service ping, why don't they use a > different port? > I think it's a security tool, but I don't know much about it. AFAIK it attempts to connect to any port in use no matter what is running there to check for problems. Cheers, Bobby |
|
From: Questions/problems r. to u. J. <jav...@li...> - 2021-11-22 08:43:46
|
On 18.11.21 13:49, Questions/problems related to using JGroups wrote: > Hi, > > One of our customers has a security tool that constantly tries to > connect to each server, so the jgroups logs have this happening every ~4 > few seconds: > > 2021-10-28 05:18:35 org.jgroups.protocols.TCP warn WARN: JGRP000006: > failed accepting connection from peer > java.io.EOFException > at java.io.DataInputStream.readFully(DataInputStream.java:197) > at > org.jgroups.blocks.cs.TcpConnection.readPeerAddress(TcpConnection.java:245) > at org.jgroups.blocks.cs.TcpConnection.<init>(TcpConnection.java:53) > at org.jgroups.blocks.cs.TcpServer$Acceptor.handleAccept(TcpServer.java:126) > at org.jgroups.blocks.cs.TcpServer$Acceptor.run(TcpServer.java:111) > at java.lang.Thread.run(Thread.java:748) > > Can that affect the performance of jgroups? I don't think so, this causes an additional TCP connection to be (half-)established, but it will be torn down immediately, so a bit of processing. If they use this as a kind of health service ping, why don't they use a different port? > I see it on all nodes, but > one of their nodes, a primary database, when under load sometimes > doesn't see view changes that the other nodes see until a minute or more > later. That should be unrelated... > Since the above happens on *every* node I'd think it's unrelated > but wanted to check. I know it's kind of a *qualitative* question, sorry. > > This is with jgroups 4.1.8.Final using a TCP stack. Can get you the full > channel creation info if it helps. > > Thanks, > Bobby > > > > _______________________________________________ > javagroups-users mailing list > jav...@li... > https://lists.sourceforge.net/lists/listinfo/javagroups-users > -- Bela Ban | http://www.jgroups.org |
|
From: Questions/problems r. to u. J. <jav...@li...> - 2021-11-18 13:46:54
|
Hi, One of our customers has a security tool that constantly tries to connect to each server, so the jgroups logs have this happening every ~4 few seconds: 2021-10-28 05:18:35 org.jgroups.protocols.TCP warn WARN: JGRP000006: failed accepting connection from peer java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at org.jgroups.blocks.cs.TcpConnection.readPeerAddress(TcpConnection.java:245) at org.jgroups.blocks.cs.TcpConnection.<init>(TcpConnection.java:53) at org.jgroups.blocks.cs.TcpServer$Acceptor.handleAccept(TcpServer.java:126) at org.jgroups.blocks.cs.TcpServer$Acceptor.run(TcpServer.java:111) at java.lang.Thread.run(Thread.java:748) Can that affect the performance of jgroups? I see it on all nodes, but one of their nodes, a primary database, when under load sometimes doesn't see view changes that the other nodes see until a minute or more later. Since the above happens on *every* node I'd think it's unrelated but wanted to check. I know it's kind of a *qualitative* question, sorry. This is with jgroups 4.1.8.Final using a TCP stack. Can get you the full channel creation info if it helps. Thanks, Bobby |