Related questions

Threshold in hierarchical clustering

2022-04-26 16:16 Eleanor Abernathy imported from Stackoverflow

I have a dataset that generates the chart attached at the end of the post. I want to cluster the data. Visually I identify 4 different clusters. As this data may change and so the number of clusters I'm using a hierarchical clustering in which I don't specify in advance the number of clusters to do.

The code

thresh = 1.00
clusters = hcluster.fclusterdata(df[['time', 'mean']].to_numpy(), thresh, criterion="distance")

Produces more than 12000 different clusters while when thresh is 1.02 (an increase of 0.02 points) gives only 1 cluster.

Can someone help to understand where is the problem in my code and how should I do in order to identify the 4 data groups that it is possible to visually observe.

enter image description here

How many English words
do you know?
Test your English vocabulary size, and measure
how many words do you know
Online Test
Powered by Examplum

See also questions close to this topic

  • Python File Tagging System does not retrieve nested dictionaries in dictionary

    I am building a file tagging system using Python. The idea is simple. Given a directory of files (and files within subdirectories), I want to filter them out using a filter input and tag those files with a word or a phrase.

    If I got the following contents in my current directory:

    data/
   budget.xls
   world_building_budget.txt
a.txt
b.exe
hello_world.dat
world_builder.spec

    and I execute the following command in the shell:

    py -3 tag_tool.py -filter=world -tag="World-Building Tool"

    My output will be:

    These files were tagged with "World-Building Tool":
data/ world_building_budget.txt
hello_world.dat
world_builder.spec

    My current output isn't exactly like this but basically, I am converting all files and files within subdirectories into a single dictionary like this:

    def fs_tree_to_dict(path_):
    file_token = ''
    for root, dirs, files in os.walk(path_):
        tree = {d: fs_tree_to_dict(os.path.join(root, d)) for d in dirs}
        tree.update({f: file_token for f in files})
        return tree

    Right now, my dictionary looks like this: key:''.

    In the following function, I am turning the empty values '' into empty lists (to hold my tags):

    def empty_str_to_list(d):
    for k,v in d.items():
        if v == '':
            d[k] = []
        elif isinstance(v, dict):
            empty_str_to_list(v)

    When I run my entire code, this is my output:

    hello_world.dat ['World-Building Tool']
world_builder.spec ['World-Building Tool']

    But it does not see data/world_building_budget.txt. This is the full dictionary:

    {'data': {'world_building_budget.txt': []},  
'a.txt': [],
'hello_world.dat': [], 
'b.exe': [], 
'world_builder.spec': []}

    This is my full code:

    import os, argparse


def fs_tree_to_dict(path_):
    file_token = ''
    for root, dirs, files in os.walk(path_):
        tree = {d: fs_tree_to_dict(os.path.join(root, d)) for d in dirs}
        tree.update({f: file_token for f in files})
        return tree


def empty_str_to_list(d):
    for k, v in d.items():
        if v == '':
            d[k] = []
        elif isinstance(v, dict):
            empty_str_to_list(v)


parser = argparse.ArgumentParser(description="Just an example", formatter_class=argparse.ArgumentDefaultsHelpFormatter)

parser.add_argument("--filter", action="store", help="keyword to filter files")
parser.add_argument("--tag", action="store", help="a tag phrase to attach to a file")
parser.add_argument("--get_tagged", action="store", help="retrieve files matching an existing tag")

args = parser.parse_args()

filter = args.filter
tag = args.tag
get_tagged = args.get_tagged
current_dir = os.getcwd()

files_dict = fs_tree_to_dict(current_dir)
empty_str_to_list(files_dict)

for k, v in files_dict.items():
    if filter in k:
        if v == []:
            v.append(tag)
            print(k, v)

        elif isinstance(v, dict):
            empty_str_to_list(v)

    if get_tagged in v:
        print(k, v)
  • Actaully i am working on a project and in it, it is showing no module name pip_internal plz help me for the same. I am using pycharm(conda interpreter 
     File "C:\Users\pjain\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\pjain\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\pjain\AppData\Local\Programs\Python\Python310\Scripts\pip.exe\__main__.py", line 4, in <module>
  File "C:\Users\pjain\AppData\Local\Programs\Python\Python310\lib\site-packages\pip\_internal\__init__.py", line 4, in <module>
    from pip_internal.utils import _log

    I am using pycharm with conda interpreter.

  • Looping the function if the input is not string

    I'm new to python (first of all) I have a homework to do a function about checking if an item exists in a dictionary or not.

    inventory = {"apple" : 50, "orange" : 50, "pineapple" : 70, "strawberry" : 30}

def check_item():
    x = input("Enter the fruit's name: ")
    if not x.isalpha():
        print("Error! You need to type the name of the fruit")
    elif x in inventory:
        print("Fruit found:", x)
        print("Inventory available:", inventory[x],"KG")
    else:
        print("Fruit not found")
check_item()

    I want the function to loop again only if the input written is not string. I've tried to type return Under print("Error! You need to type the name of the fruit") but didn't work. Help

  • Why KMedoids and Hierarchical return different results?

    I have a huge dataframe which only contains 0 and 1, and I tried to use the method scipy.cluster.hierarchy to get the dendrogram and then use the method sch.fcluster to get the cluster by a specific cutoff. (the metric for distance matrix is Jacccard, the method for linkage is "centroid")

    However, when I want to specify the optimistic numbers of clusters for my dataframe, I notice the method of KMedoids combined with the Elbow Method can help me. Then after I know the best numbers of clusters such as 2, I tried to use KMedoids(n_clusters=2,metric='jaccard').fit(dataset) to get clusters, but the result is different from Hierarchical method. (the reason why I don't use Kmeans is that it is too slow for my dataframe)

    Therfore, I did a test (the index 0,1,2,3 will be grouped):

    import pandas as pd
import numpy as np
from scipy.spatial.distance import pdist

label1 = np.random.choice([0, 1], size=20)
label2 = np.random.choice([0, 1], size=20)
label3 = np.random.choice([0, 1], size=20)
label4 = np.random.choice([0, 1], size=20)
dataset = pd.DataFrame([label1,label2,label3,label4])
dataset

    enter image description here

    Method KMedoids:

    since there only are 4 indexes, so the cluster number was set to 2.

    from sklearn_extra.cluster import KMedoids
cobj = KMedoids(n_clusters=2,metric='jaccard').fit(dataset)
labels = cobj.labels_
labels

    the clustering result as shown below:

    enter image description here

    Method Hierarchical:

    import scipy.cluster.hierarchy as such
#calculate distance matrix
disMat = sch.distance.pdist(dataset, metric='jaccard')
disMat1 = sch.distance.squareform(disMat)
# cluster:
Z2=sch.linkage(disMat1,method='centroid')
sch.fcluster(Z2, t=1, criterion='distance')

    to meet the same number of clusters I tried several cutoff, the number of cluster was 2 when the cutoff was set to 1. Here is the result:

    enter image description here

    And I googled about the dataframe which was passed to KMedoids should be the original dataframe, not the distance matrix. but it seems that KMedoids will convert the original dataframe to a new one which I don't know for some reason. because I got the data conversion warning:

    DataConversionWarning: Data was converted to boolean for metric jaccard
  warnings.warn(msg, DataConversionWarning)

    I also got warning when I perform Hierarchical method:

    ClusterWarning: scipy.cluster: The symmetric non-negative hollow observation matrix looks suspiciously like an uncondensed distance matrix

    Purpose:

    What I want is to find some method to get the clusters if I know the optimal number of clusters. but the method Hierarchical need to try different cutoff, while the KMedoids don't, but it turns a different result.

    Can anybody explain this to me? And are there better ways to perform clustering?

  • R: Double Clustering of Standard Errors in Panel Regression

    so i am analysing fund data. I use a fixed effect model and want to double cluster my standard errors along "ISIN" and "Date" with plm().

    output for dput(data) is :

    > dput(nd[1:100, ])
structure(list(Date = structure(c(1517356800, 1519776000, 1522454400, 
1525046400, 1527724800, 1530316800, 1532995200, 1535673600, 1538265600, 
1540944000, 1543536000, 1546214400, 1548892800, 1551312000, 1553990400, 
1556582400, 1559260800, 1561852800, 1564531200, 1567209600, 1569801600, 
1572480000, 1575072000, 1577750400, 1580428800, 1582934400, 1585612800, 
1588204800, 1590883200, 1593475200, 1596153600, 1598832000, 1601424000, 
1604102400, 1606694400, 1609372800, 1612051200, 1614470400, 1617148800, 
1619740800, 1622419200, 1625011200, 1627689600, 1630368000, 1632960000, 
1635638400, 1638230400, 1640908800, 1517356800, 1519776000, 1522454400, 
1525046400, 1527724800, 1530316800, 1532995200, 1535673600, 1538265600, 
1540944000, 1543536000, 1546214400, 1548892800, 1551312000, 1553990400, 
1556582400, 1559260800, 1561852800, 1564531200, 1567209600, 1569801600, 
1572480000, 1575072000, 1577750400, 1580428800, 1582934400, 1585612800, 
1588204800, 1590883200, 1593475200, 1596153600, 1598832000, 1601424000, 
1604102400, 1606694400, 1609372800, 1612051200, 1614470400, 1617148800, 
1619740800, 1622419200, 1625011200, 1627689600, 1630368000, 1632960000, 
1635638400, 1638230400, 1640908800, 1517356800, 1519776000, 1522454400, 
1525046400), tzone = "UTC", class = c("POSIXct", "POSIXt")), 
    Dummy = c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 
    0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 
    1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 
    1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 
    1, 1, 1, 1, 0, 0, 0, 0), ISIN = c("LU1883312628", "LU1883312628", 
    "LU1883312628", "LU1883312628", "LU1883312628", "LU1883312628", 
    "LU1883312628", "LU1883312628", "LU1883312628", "LU1883312628", 
    "LU1883312628", "LU1883312628", "LU1883312628", "LU1883312628", 
    "LU1883312628", "LU1883312628", "LU1883312628", "LU1883312628", 
    "LU1883312628", "LU1883312628", "LU1883312628", "LU1883312628", 
    "LU1883312628", "LU1883312628", "LU1883312628", "LU1883312628", 
    "LU1883312628", "LU1883312628", "LU1883312628", "LU1883312628", 
    "LU1883312628", "LU1883312628", "LU1883312628", "LU1883312628", 
    "LU1883312628", "LU1883312628", "LU1883312628", "LU1883312628", 
    "LU1883312628", "LU1883312628", "LU1883312628", "LU1883312628", 
    "LU1883312628", "LU1883312628", "LU1883312628", "LU1883312628", 
    "LU1883312628", "LU1883312628", "NL0000289783", "NL0000289783", 
    "NL0000289783", "NL0000289783", "NL0000289783", "NL0000289783", 
    "NL0000289783", "NL0000289783", "NL0000289783", "NL0000289783", 
    "NL0000289783", "NL0000289783", "NL0000289783", "NL0000289783", 
    "NL0000289783", "NL0000289783", "NL0000289783", "NL0000289783", 
    "NL0000289783", "NL0000289783", "NL0000289783", "NL0000289783", 
    "NL0000289783", "NL0000289783", "NL0000289783", "NL0000289783", 
    "NL0000289783", "NL0000289783", "NL0000289783", "NL0000289783", 
    "NL0000289783", "NL0000289783", "NL0000289783", "NL0000289783", 
    "NL0000289783", "NL0000289783", "NL0000289783", "NL0000289783", 
    "NL0000289783", "NL0000289783", "NL0000289783", "NL0000289783", 
    "NL0000289783", "NL0000289783", "NL0000289783", "NL0000289783", 
    "NL0000289783", "NL0000289783", "DE0008474008", "DE0008474008", 
    "DE0008474008", "DE0008474008"), Returns = c(-0.12401, -4.15496, 
    -1.39621, 4.46431, -2.28814, -0.58213, 3.61322, -3.56401, 
    0.6093, -4.73124, 0.88597, -5.55014, 5.12313, 2.65441, 1.3072, 
    2.99972, -5.1075, 3.51965, 0.24626, -2.21961, 4.48332, -0.03193, 
    2.19313, 1.81355, -2.2836, -8.3185, -14.58921, 4.47981, 4.52948, 
    5.51294, -2.16857, 2.56992, -2.04736, -6.17825, 14.71218, 
    1.24079, -1.33888, 3.5197, 8.09674, 1.43074, 3.79434, 0.47398, 
    1.57474, 2.48837, -3.08439, 3.68851, -2.93803, 6.43656, 2.67598, 
    -3.39767, -5.27997, 4.76756, 4.89914, -0.95931, 2.22484, 
    3.01478, 1.63997, -6.64158, 3.46497, -8.54853, 7.40113, 5.68973, 
    1.64367, 4.35256, -5.09351, 3.43618, 2.16774, -0.77703, 3.16832, 
    1.65626, 4.91897, 1.76163, 1.49508, -5.16847, -9.53639, 12.74246, 
    3.08746, 3.4028, 0.09515, 5.66077, -2.85661, -2.58972, 9.53565, 
    2.93138, 0.32556, 2.92393, 5.02059, 0.98137, 0.58733, 4.91219, 
    2.21603, 2.52087, -3.87762, 7.66159, -0.04559, 4.48257, 2.83511, 
    -6.27841, -3.98683, 4.99554), Flows = c(-0.312598458, -37.228563578, 
    -119.065088084, -85.601069424, -46.613436838, -20.996760878, 
    -12.075112555, -40.571568112, -16.210315254, -54.785115578, 
    -55.93565336, -25.073939479, -16.513305702, -111.112262813, 
    -17.260252326, -44.287088276, -84.358676293, -12.73665543, 
    -14.846322594, -30.353217826, -43.002634628, -31.293725624, 
    -32.291532262, -21.145334594, -33.460150254, -22.458849454, 
    -34.690817528, -34.088358344, -4.069613214, -7.841523244, 
    -6.883674001, -11.99060429, -19.155102931, -20.274682083, 
    -33.509645025, -25.764368282, -22.451403457, -39.075362392, 
    -9.772306537, -7.214728071, -10.462230506, -12.550102699, 
    -0.439609898, -16.527865041, -15.938402293, -10.916678964, 
    -11.041205907, -11.627537098, -13.797947969, -18.096144272, 
    29.879529566, -51.895196556, -3.192064966, -1.469562773, 
    9.739671656, -35.108549922, -19.490401121, 36.459406559, 
    -66.213269625, 8.105824198, -17.078089399, -59.408458411, 
    1.227033593, -42.501421101, -15.275983037, 19.425363714, 
    -23.165013159, -19.68599313, -20.478530269, -19.566890333, 
    -19.63229278, -59.274372862, -37.128708445, 5.129404763, 
    -2.650978954, -0.566245645, -14.80700799, 4.891308881, -18.16286654, 
    -17.570559084, -2.726629634, -14.482219321, -35.795673521, 
    -10.119935801, -14.37900783, -20.385053784, -4.550848701, 
    -17.672355509, -14.270420088, 1.440911458, -8.924636198, 
    -5.749771862, -12.284920947, -23.093834986, -13.553880939, 
    -31.572182943, -22.977082191, -8.076560195, -11.825577374, 
    -9.263872938), TNA = c(2474.657473412, 2327.75517961, 2171.146502197, 
    2175.433117247, 2082.147188171, 2042.121760963, 2031.311390907, 
    1918.904748403, 1914.140451001, 1765.867322561, 1724.972362171, 
    1600.059421422, 1605.009162592, 1539.205393073, 1540.8291693, 
    1538.550310809, 1370.631945404, 1404.091772234, 1351.60138448, 
    1290.98574898, 1309.942298579, 1280.634128059, 1278.146819041, 
    1281.50075434, 1189.563983023, 1062.001168646, 859.735053702, 
    868.096185968, 894.397805491, 933.614731653, 885.975121845, 
    897.018097461, 854.196359787, 781.178047528, 863.00585297, 
    846.859512502, 796.10866733, 784.290994645, 838.747509395, 
    841.511540715, 863.678978862, 854.663205271, 856.363306246, 
    859.460891875, 816.275861034, 836.347760358, 800.867957871, 
    842.657752288, 2742.709413, 2629.70296, 2518.690562, 2516.902480001, 
    2635.037923, 2606.124805, 2672.082125, 2715.556617, 2738.845915, 
    2591.318371, 2613.260789, 2396.060545001, 2554.437804, 2638.160519, 
    2680.990319, 2753.467368, 2533.347075001, 2637.887076, 2670.127393, 
    2628.138778001, 2688.643794, 2711.56785, 2823.634535001, 
    2811.983963001, 2835.218976, 2672.765021, 2413.332814, 2718.586512, 
    2727.69596, 2823.040628, 2805.482839, 2944.602701, 2855.870812, 
    2765.189256, 2990.804719, 3066.36598, 3059.603769, 3126.458368, 
    3276.612153, 3289.257788, 3291.864476, 3397.759970999, 3461.462599, 
    3540.518638, 3388.702548, 3622.641661, 3604.82519, 3732.115875999, 
    4129.617979, 3857.780349, 3687.848268001, 3858.323607), Age = c(2, 
    2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 
    3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 5, 5, 5, 
    5, 5, 5, 5, 5, 5, 5, 5, 5, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 
    9, 9, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 11, 
    11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 12, 12, 12, 12, 
    12, 12, 12, 12, 12, 12, 12, 12, 62, 62, 62, 62)), row.names = c(NA, 
-100L), class = c("tbl_df", "tbl", "data.frame"))

    My code did yield me initially a result, i didn't change anything but all of the sudden it doesn't allow me to execute the last line of code.

    library(plm)
attach(nd)
library(lmtest)
library(stargazer)
library(sandwich)
library(etable)
library(pacman)
library(fixest)
library(multiwayvcov)
library(foreign)


#cleaning
#adjust units of TNA and Flows 
nd <- nd %>% 
  mutate(TNA = TNA / 1000000, Flows = Flows / 1000000) #1mio and 1mio 
#drop na's
#nd <- nd %>%
  #drop_na()
#variable creation for model
Y <- cbind(nd$Flows)
X <- cbind(nd$Dummy, lag(nd$Returns), lag(nd$TNA), nd$Age) 

# descriptive statistics 
summary(Y)
summary(X)

#random effects 
        random2 <- plm(Y ~ X, nd, model='random', index=c('ISIN', 'Date'))
        summary(random2)

#fixed effect model 
fixed2 <- plm(Y ~ X, nd, model='within', index=c('ISIN', 'Date'))

# Breusch-Pagan Test 
bptest(fixed2)

#Test which model to use fixed effect or random effects
#hausmann test 
phtest(random2, fixed2) # we take fixed effects 

##Double-clustering formula (Thompson, 2011)
vcovDC <- function(x, ...){
  vcovHC(x, cluster="ISIN", ...) + vcovHC(x, cluster="Date", ...) - 
    vcovHC(x, method="white1", ...)
}

#visualize SEs
coeftest(fixed2, vcov=function(x) vcovDC(x, type="HC1"))
stargazer(coeftest(fixed2, vcov=function(x) vcovDC(x, type="HC1")), type = "text")

    Now, when i try to run:

    coeftest(fixed2, vcov=function(x) vcovDC(x, type="HC1"))

    I get the error: Error in match.arg(cluster) : 'arg' should be one of “group”, “time” Before it didn't.

    I highly appreciate any answer. I'd also like to know if the formula i used for the double clustered standard errors is correct. I followed the approach from: Double clustered standard errors for panel data

    • the comment from Iandorin

    edit: i rewrote the code and now it works:

    library(plm)
attach(nd)
library(lmtest)
library(stargazer)
library(sandwich)
library(etable)
library(pacman)
library(fixest)
library(multiwayvcov)
library(foreign)


#cleaning
#adjust units of TNA and Flows 
#nd <- nd %>% 
  #mutate(TNA = TNA / 1000000, Flows = Flows / 1000000) #1mio and 1mio 
#drop na's
#nd <- nd %>%
  #drop_na()

#variable creation for model
Y <- cbind(nd$Flows)
X <- cbind(nd$Dummy, lag(nd$Returns), lag(nd$TNA), nd$Age) 

# descriptive statistics 
summary(Y)
summary(X)


#random effects 
        random2 <- plm(Y ~ X, nd, model='random', index=c('ISIN', 'Date'))
        summary(random2)

#fixed effect model 
fixed2 <- plm(Y ~ X, nd, model='within', index=c('ISIN', 'Date'))

# Breusch-Pagan Test 
bptest(fixed2)

#Test which model to use fixed effect or random effects
#hausmann test 
phtest(random2, fixed2) # we take fixed effects 



##Double-clustering formula (Thompson, 2011)
vcovDC <- function(x, ...){
  vcovHC(x, cluster="ISIN", ...) + vcovHC(x, cluster="Date", ...) - 
    vcovHC(x, method="white1", ...)
}

testamk <- plm(Y ~ X, nd, model='within', index=c('ISIN', 'Date'))
summary(testamk)

coeftest(testamk, vcov=function(x) vcovHC(x, cluster="group", type="HC1"))

    Many thanks in advance! Joe

  • Seurat - cannot plot the same dimplot again

    I am trying to rewrite the code of this paper: https://doi.org/10.1038/s42003-020-0837-0
    I have written the code step-by-step based on the instructions mentioned in the methods section. But after clustering, for plotting the clusters by dimplot, I receive a dissimilar plot compared to the same plot in the paper.
    I wonder what is the problem? I have tailored every parameter to receive the same plot but it hasn't worked yet.
    Graph of the paper
    My graph
    Please help me to solve this issue.

  • R quickpsy calculating JNDs

    I'm using quickpsy in R http://dlinares.org/quickpsy.html to calculate thresholds. Could someone help me extract JNDs from quickpsy calculations?

    Example task: https://rstudio-pubs-static.s3.amazonaws.com/446272_5aa2c51c9e2d4e71b2cf8229e205ce64.html hier in the end one is asked to extract and visualise JNDs.

    enter image description here

    I'm using quickpsy function to make estimations. In the image above you can see all the estimations quickpsy is calculating. How do I extract JNDs? I tried comparing thresholds at 0.75% and 0.50% probabilities. I'm not sure whether I'm getting accurate JND values that way.

  • Imbalanced Binary Dataset in Keras. Finding the best threshold after fit s.t. sensitivity and specificity is maximized?

    I have made an ANN in Keras that works on an imbalanced binary dataset. The data is used after fitting the model to predict the binary classes and I want to choose a threshold s.t. sensitivity and specificity is maximized.

    This is the code that I am using right now, iterating through all thresholds from 0-1 and finding the best one using G-mean score. But is there a better way to maximize sens and spec? Perhaps finding a sens and spec suchs that

    | sens - spec | < 0.05 and sens*spec > score_max

    Then once this score_max is found you can run through smaller jumps for like +- 0.2 on both? Or is there another way to find sensitivity and specificity maximum?

    predictions = model_p.predict(Xt)

    #predicts = model_p.predict_proba(Xt)


    thresholds = arange(0, 1, 0.001)
    threshold = -1
    best_Gscore = 0

    


    false_positive = 0
    true_positive = 0
    false_negative = 0
    true_negative = 0
    # Show all cases
    for z in thresholds:
        print("Threshold => %f " % (z))
        fp = 0
        fn = 0
        tp = 0
        tn = 0
        for i in range(len(yt)):
            if( yt[i] == 0 and predictions[i] > z ):
                fp += 1
            elif( yt[i] == 1 and predictions[i] > z ):
                tp += 1
            elif( yt[i] == 1 and predictions[i] <= z ):
                fn += 1
            elif( yt[i] == 0 and predictions[i] <= z ):
                tn += 1
        
        if( (tp+fn) == 0):
            continue
        if( (tn+fp) == 0):
            continue
        TPR = fp / (fp + tn)
        #sens = tp / (tp + fn)
        #spec = tn / (tn + fp)
        FPR = tp / (tp + fn)
        Gscore = math.sqrt(TPR*(1-FPR))

        print("J Stat => %f " % (Gscore), flush=True)

        if( Gscore > best_Gscore ):
            best_Gscore = Gscore
            false_positive = fp
            false_negative = fn
            true_positive = tp
            true_negative = tn
            threshold = z
  • Defining metric and threshold values for one of the sub-characteristic from ISO 25010 for a add new member" function

    May I know is there any paper or research explaining how to set the metric and threshold of the "add new member" function. is there any similar research that can be referred?

  • Search for ancestors in a hierarchical list

    Created a method to search for children in a hierarchical list. Is there a way to do the method over the stack?

    public static IEnumerable<T> GetSelfAndAncestors<T>(this T item, Func<T, T> parentSelector)
{
    var parent = parentSelector(item);
    if (parent==null)
        return item.ToIEnumerable();

    return item.ToIEnumerable().Concat(parent.GetSelfAndAncestors(parentSelector));
}
  • How to get the current substate and the parent state out of the Spring Statemachine?

    I am running a hierachical Spring Statemachine and - after walking through the inital transitions into state UP with the default substate STOPPED - want to use statemachine.getState(). Trouble is, it gives me only the parent state UP, and I cannot find an obvious way to retrieve both the parent state and the sub state.

    The machine has states constructed like so:

        StateMachineBuilder.Builder<ToolStates, ToolEvents> builder = StateMachineBuilder.builder();


    builder.configureStates()
       .withStates()
          .initial(ToolStates.UP)
          .state(ToolStates.UP, new ToolUpEventAction(), null)
          .state(ToolStates.DOWN                
          .and()
       .withStates()
          .parent(ToolStates.UP)
          .initial(ToolStates.STOPPED)
          .state(ToolStates.STOPPED,new ToolStoppedEventAction(), null )
          .state(ToolStates.IDLE)
          .state(ToolStates.PROCESSING,
                 new ToolBeginProcessingPartAction(),
                 new ToolDoneProcessingPartAction());

    ...

    builder.build();

    ToolStates and ToolEvents are just enums. In the client class, after running the builder code above, the statemachine is started with statemachine.start(); When I subsequently call statemachine.getState().getId(); it gives me UP. No events sent to statemachine before that call. I have been up and down the Spring statemachine docs and examples. I know from debugging that the entry actions of both states UP and STOPPED have been invoked, so I am assuming they are both "active" and would want to have both states presented when querying the statemachine. Is there a clean way to achieve this ? I want to avoid storing the substate somewhere from inside the Action classes, since I believe I have delegated all state management issues to the freakin Statemachine in the first place and I would rather like to learn how to use its API for this purpose.

    Hopefully this is something embarrasingly obvious...

    Any advice most welcome!