How to apply the learned model using boosting technique

2018-06-22 21:19 Srishti M imported from Stackoverflow

I have no clue on how to apply the learned model using adaptive boosting method since there is not model as such that is returned similar to the one returned by say the SVM classifier's fitcsvm() function. I am using the implementation given https://www.mathworks.com/matlabcentral/fileexchange/63162-adaboost?focused=7722161&tab=function

This method uses 5 base classifiers and so the number of learning cycles =5. Variable ada_test contains the labels from each independent run, itr of the boosting module.

QUESTION: For the boosting method once the training is done which includes the training set and prediction on the validation set, which variable should be used to predict on the unseen data set for testing? In the following code,

for itr=1:maxItr
    [~,ada_test(:,itr)]= adaboost(X,Y, Xtest);
    fm_=[fm_; confusion_mat(Ytest, ada_test(:,itr))];
end

the variable ada_test(:,1)=sign(Htest*alpha'); are the predicted labels. In the function, adaboost() there is the line ada_train(:,1)=sign(H*alpha'); Why ada_train variable is never used and how to test the learned adaboost model on unseen data? I cannot understand the procedure to predict on a new unseen test data because I cannot follow which variable or what is the learned model. Please help.

See also questions close to this topic

How to setup byte length code for data transfer over TCPIP

After conversations about TCPIP communication, it was brought to my attention I should be sending byte lengths in front of my packets to notify the server how much data is being sent. The code I currently have is as follows:

Python Client

import json
import time
import multiprocessing as mp
import socket
import os

class ClientBuddy():
    def __init__(self):
        with open('config2.json', 'r') as jfile:
            confjson = json.load(jfile)     
        asyncore.dispatcher.__init__(self)                                      # Initialize asyncore
        self.TransferSocket = socket.socket(socket.AF_INET,socket.SOCK_STREAM)  # create socket connection
        self.host = confjson.get('HOST')                                                        # set host name as defined above
        self.port = confjson.get('PORT')                                                        # set port name as defined above
        self.TransferSocket.connect((self.host, self.port))                     # establish connection to server
        self.filename = "Heybud.h264"

    def Openport(self):
        print("Start Transfer")
        FName = self.filename                                                   # Put is a server side command to place file somewhere on computer
        len(FName)
        msg = []
        self.TransferSocket.send(FName.encode())                                # send file name over to computer
        f = open(self.filename,"rb")
        chunk = f.read(1024)
        while(chunk):
            self.TransferSocket.send(chunk)
            if self.TransferSocket.recv(1024) != self.AckFl.encode():
                print("Failed transfer")
            chunk = f.read(1024)
        f.close()
        print("Transfer Complete")

    def Senddata(self):
        runTransfer = mp.Process(target=self.Openport())
        runTransfer.start()
        runTransfer.join()

ClientBuddy().Senddata()

No clue where to begin with writing the byte lengths. I thought of using len to find the length of the file name and than converting to a string or binary.

Matlab Server: (Currently setup for 1 connection but plan to have multiple connections later)

classdef ServersideCom<handle
    properties
        ClientIP = '0.0.0.0';
        Port = 5650;
        TCPIPServer
    end

    methods
        function gh = ServersideCom(~)
             poolobj = gcp('nocreate');
             if isempty(poolobj)
                 gh.myPool = parpool('IdleTimeout', 120);
             end
        end
        function SetupServer(gh,connections)
           for aa = 1:connections
               cIP = gh.clientIP;
               Pts = gh.Port-2;
               ts{aa} = tcpip(cIP,(Pts+(aa*2)),'NetworkRole','Server');
               set(gh.tcpipServer,'OutputBufferSize',gh.bsize);
               fopen(ts{aa});
               display('Established Connection')
           end
           gh.tcpipServer = ts;
        end

        function Serverpull(gh,Port)
            gh.msg = fread(gh.tcpipServer{Port});
            display(gh.msg) 
        end

I am in the process of writing the Data acquisition side of the server, hence why it only displays the packages. Now at this point. How should I go about writing a byte length and sending to the server. I'm guessing I should have in the matlab side

code = gh.msg(1:4) % since the codes going to be 4 digits long
Bitpull = str2double(code);
Filename = gh.msg(5:Bitpull);
code = gh.msg(Bitpull+1:Bitpull+5);
....

However this feels like it could be prone to errors (though I could be wrong)

I was told this is low level sockets so If a higher level socket is available and could eliminate having to write code like this, I'm for it. I however am only aware of this right now so this is what I'm trying to do.

Retrieve data point before intersection with a polygon in matlab

I am very new to Matlab. Shown is a visual of my problem. box 1 and 2

I already have the intersection points i (taken from polyxpoly) now what's needed is the datapoint in matrix c before the intersection.

Using knnsearch would always return the nearest neighbor. In the first box, knnsearch would return point B, which is correct. In the second box, it would also return point B but I needed it to return point A.

I thought of inserting the intersection points to c but I can't think of a way how to index it.
what should i do to make alexnet not detect whatever is not required ?

i am using alexnet for image classification using webcam in matlab. i have made 5 different sub folders with various sample images of different objects like pen,book,chair,mouse,mobile. Now my problem is, if i show any of these objects to the webcam, it detects accurately. if the webcam is randomly pointed to any other object like a keyboard, it still gives me name of some sub folder i have created. what should i do to make alexnet not detect whatever is not required ? i am not a computers student and im not so good with coding.

Shape Error in Andrew NG Logistic Regression using Scipy.opt

I've been trying to write Andrew NG's Logistic Regression Problem Using python and Scipy.opt for optimizing the function. However, I get a VALUE ERROR that says I have mismatching dimensions. I've tried to flatten() my theta array as scipy.opt doesn't seem to work very well with single column/row vector, however the problem still persists.

Kindly point me in the right direction as to what is causing the problem and how to avoid it.

Thanks a million!

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import scipy.optimize as opt

dataset = pd.read_csv("Students Exam Dataset.txt", names=["Exam 1", "Exam 2", "Admitted"])
print(dataset.head())

positive = dataset[dataset["Admitted"] == 1]
negative = dataset[dataset["Admitted"] == 0]

#Visualizing Dataset
plt.scatter(positive["Exam 1"], positive["Exam 2"], color="blue", marker="o", label="Admitted")
plt.scatter(negative["Exam 1"], negative["Exam 2"], color="red", marker="x", label="Not Admitted")
plt.xlabel("Exam 1 Score")
plt.ylabel("Exam 2 Score")
plt.title("Admission Graph")
plt.legend()
#plt.show()

#Preprocessing Data
dataset.insert(0, "x0", 1)
col = len(dataset.columns)
x = dataset.iloc[:,0:col-1].values
y = dataset.iloc[:,col-1:col].values
b = np.zeros([1,col-1])
m = len(y)
print(f"X Shape: {x.shape}   Y Shape: {y.shape}   B Shape: {b.shape}")

#Defining Functions
def hypothesis(x, y, b):
    h = 1 / (1+np.exp(-x @ b.T))
    return h

def cost(x, y, b):
    first = (y.T @ np.log(hypothesis(x, y, b)))
    second = (1-y).T @ np.log(1 - hypothesis(x, y, b))
    j = (-1/m) * np.sum(first+second)
    return j

def gradient(x, y, b):
    grad_step = ((hypothesis(x, y, b) - y) @ x.T) / m
    return b

#Output
initial_cost = cost(x, y, b)
print(f"\nInitial Cost = {initial_cost}")
final_cost = opt.fmin_tnc(func=cost, x0=b.flatten() , fprime=gradient, args=(x,y))
print(f"Final Cost = {final_cost} \nTheta = {b}")

Dataset Used: ex2.txt

34.62365962451697,78.0246928153624,0
30.28671076822607,43.89499752400101,0
35.84740876993872,72.90219802708364,0
60.18259938620976,86.30855209546826,1
79.0327360507101,75.3443764369103,1
45.08327747668339,56.3163717815305,0
61.10666453684766,96.51142588489624,1
75.02474556738889,46.55401354116538,1
76.09878670226257,87.42056971926803,1
84.43281996120035,43.53339331072109,1
95.86155507093572,38.22527805795094,0
75.01365838958247,30.60326323428011,0
82.30705337399482,76.48196330235604,1
69.36458875970939,97.71869196188608,1
39.53833914367223,76.03681085115882,0
53.9710521485623,89.20735013750205,1
69.07014406283025,52.74046973016765,1
67.94685547711617,46.67857410673128,0
70.66150955499435,92.92713789364831,1
76.97878372747498,47.57596364975532,1
67.37202754570876,42.83843832029179,0
89.67677575072079,65.79936592745237,1
50.534788289883,48.85581152764205,0
34.21206097786789,44.20952859866288,0
77.9240914545704,68.9723599933059,1
62.27101367004632,69.95445795447587,1
80.1901807509566,44.82162893218353,1
93.114388797442,38.80067033713209,0
61.83020602312595,50.25610789244621,0
38.78580379679423,64.99568095539578,0
61.379289447425,72.80788731317097,1
85.40451939411645,57.05198397627122,1
52.10797973193984,63.12762376881715,0
52.04540476831827,69.43286012045222,1
40.23689373545111,71.16774802184875,0
54.63510555424817,52.21388588061123,0
33.91550010906887,98.86943574220611,0
64.17698887494485,80.90806058670817,1
74.78925295941542,41.57341522824434,0
34.1836400264419,75.2377203360134,0
83.90239366249155,56.30804621605327,1
51.54772026906181,46.85629026349976,0
94.44336776917852,65.56892160559052,1
82.36875375713919,40.61825515970618,0
51.04775177128865,45.82270145776001,0
62.22267576120188,52.06099194836679,0
77.19303492601364,70.45820000180959,1
97.77159928000232,86.7278223300282,1
62.07306379667647,96.76882412413983,1
91.56497449807442,88.69629254546599,1
79.94481794066932,74.16311935043758,1
99.2725269292572,60.99903099844988,1
90.54671411399852,43.39060180650027,1
34.52451385320009,60.39634245837173,0
50.2864961189907,49.80453881323059,0
49.58667721632031,59.80895099453265,0
97.64563396007767,68.86157272420604,1
32.57720016809309,95.59854761387875,0
74.24869136721598,69.82457122657193,1
71.79646205863379,78.45356224515052,1
75.3956114656803,85.75993667331619,1
35.28611281526193,47.02051394723416,0
56.25381749711624,39.26147251058019,0
30.05882244669796,49.59297386723685,0
44.66826172480893,66.45008614558913,0
66.56089447242954,41.09209807936973,0
40.45755098375164,97.53518548909936,1
49.07256321908844,51.88321182073966,0
80.27957401466998,92.11606081344084,1
66.74671856944039,60.99139402740988,1
32.72283304060323,43.30717306430063,0
64.0393204150601,78.03168802018232,1
72.34649422579923,96.22759296761404,1
60.45788573918959,73.09499809758037,1
58.84095621726802,75.85844831279042,1
99.82785779692128,72.36925193383885,1
47.26426910848174,88.47586499559782,1
50.45815980285988,75.80985952982456,1
60.45555629271532,42.50840943572217,0
82.22666157785568,42.71987853716458,0
88.9138964166533,69.80378889835472,1
94.83450672430196,45.69430680250754,1
67.31925746917527,66.58935317747915,1
57.23870631569862,59.51428198012956,1
80.36675600171273,90.96014789746954,1
68.46852178591112,85.59430710452014,1
42.0754545384731,78.84478600148043,0
75.47770200533905,90.42453899753964,1
78.63542434898018,96.64742716885644,1
52.34800398794107,60.76950525602592,0
94.09433112516793,77.15910509073893,1
90.44855097096364,87.50879176484702,1
55.48216114069585,35.57070347228866,0
74.49269241843041,84.84513684930135,1
89.84580670720979,45.35828361091658,1
83.48916274498238,48.38028579728175,1
42.2617008099817,87.10385094025457,1
99.31500880510394,68.77540947206617,1
55.34001756003703,64.9319380069486,1
74.77589300092767,89.52981289513276,1

Are there pre-trained model to recognize character?

I extracted from number-plate into individual characters as picture.

https://i.stack.imgur.com/k09UB.jpg

I wonder there are pre-trained CNN model to this recognize characters. If any, please recommend for me API to perform this. Thanks a lot.
How to tune hyper parameters of LightGBM using bayesian optimization with rmse as metric

I'm trying to tune hyperparameters of lightgbm model with rmse as metric. lgbBO = BayesianOptimization(lgb_eval, {'num_leaves': (24, 45), 'feature_fraction': (0.1, 0.9), 'bagging_fraction': (0.8, 1), 'max_depth': (5, 8.99), 'lambda_l1': (0, 5), 'lambda_l2': (0, 3), 'min_split_gain': (0.001, 0.1), 'min_child_weight': (5, 50)}, random_state=0) after this what should i do?

DCGAN Tensorflow code doesn't produce faces on CelebA dataset

I have written the following code but does not produce faces on the celebA dataset. I think it should create some sort of face (even if very blurry) at the last iteration of each epoch. However, it just creates noisy squares with no visible face. I am quite new to GANs and I am not sure how to debug this Deep Convolutional GAN (DCGAN) in order to figure what's going wrong.

My code might be easier to be seen here: https://pastebin.com/c4QUqxJy Here is the code:

from __future__ import print_function
import random
import os
import glob
import scipy

import tensorflow as tf
import numpy as np
from PIL import Image
import skimage.io as io
import matplotlib.pyplot as plt


class Arguments(object):

    data_path = 'results_celebA/preprocessed/'
    save_path = 'results_celebA'                           #path to save preprocessed image folder
    preproc_foldername = 'preprocessed'      #folder name for preprocessed images
    image_size = 64                          #images are resized to image_size value 
    num_images = 202590                      #the number of training images
    batch_size = 64                          #batch size
    dim_z = 100                              #the dimension of z variable (the generator input dimension)        
    n_g_filters = 64                         #the number of the generator filters (gets multiplied between layers)
    n_f_filters = 64                         #the number of the discriminator filters (gets multiplied between layers)           
    n_epoch = 25                             #the number of epochs
    lr = 0.0002                              #learning rate
    beta1 = 0.5                              #beta_1 parameter of Adam optimizer
    beta2 = 0.99                             #beta_2 parameter of Adam optimizer

args = Arguments()


#contains functions that load, preprocess and visualize images. 


class Dataset(object):     
    def __init__(self, data_path, num_imgs, target_imgsize):
        self.data_path = data_path
        self.num_imgs = num_imgs 
        self.target_imgsize = target_imgsize 

    def normalize_np_image(self, image):
        return (image / 255.0 - 0.5) / 0.5

    def denormalize_np_image(self, image):
        return (image * 0.5 + 0.5) * 255

    def get_input(self, image_path):
        image = np.array(Image.open(image_path)).astype(np.float32)
        return self.normalize_np_image(image)

    def get_imagelist(self, data_path, celebA=False): 
        if celebA == True:
            imgs_path = os.path.join(data_path, 'img_align_celeba/*.jpg')
        else:
            imgs_path = os.path.join(data_path, '*.jpg') 
        all_namelist = glob.glob(imgs_path, recursive=True)
        return all_namelist[:self.num_imgs]

    def load_and_preprocess_image(self, image_path): 
        image = Image.open(image_path)
        j = (image.size[0] - 100) // 2
        i = (image.size[1] - 100) // 2
        image = image.crop([j, i, j + 100, i + 100])    
        image = image.resize([self.target_imgsize, self.target_imgsize], Image.BILINEAR)
        image = np.array(image.convert('RGB')).astype(np.float32)
        image = self.normalize_np_image(image)
        return image    

    #reads data, preprocesses and saves to another folder with the given path. 
    def preprocess_and_save_images(self, dir_name, save_path=''): 
        preproc_folder_path = os.path.join(save_path, dir_name)
        if not os.path.exists(preproc_folder_path):
            os.makedirs(preproc_folder_path)   
            imgs_path = os.path.join(self.data_path, 'img_align_celeba/*.jpg')
            print('Saving and preprocessing images ...')
            for num, imgname in enumerate(glob.iglob(imgs_path, recursive=True)):
                cur_image = self.load_and_preprocess_image(imgname)
                cur_image = Image.fromarray(np.uint8(self.denormalize_np_image(cur_image)))
                cur_image.save(preproc_folder_path + '/preprocessed_image_%d.jpg' %(num)) 
        self.data_path= preproc_folder_path

    def get_nextbatch(self, batch_size):
        print("nextbatch batchsize is: ", batch_size) 
        assert (batch_size > 0),"Give a valid batch size"
        cur_idx = 0
        image_namelist = self.get_imagelist(self.data_path)
        while cur_idx + batch_size <= self.num_imgs:
            cur_namelist = image_namelist[cur_idx:cur_idx + batch_size]
            cur_batch = [self.get_input(image_path) for image_path in cur_namelist]
            cur_batch = np.array(cur_batch).astype(np.float32)
            cur_idx += batch_size
            yield cur_batch

    def show_image(self, image, normalized=True):
        if not type(image).__module__ == np.__name__:
            image = image.numpy()
        if normalized:
            npimg = (image * 0.5) + 0.5 
        npimg.astype(np.uint8)
        plt.imshow(npimg, interpolation='nearest')


#contains functions that load, preprocess and visualize images. 

class Dataset(object):     
    def __init__(self, data_path, num_imgs, target_imgsize):
        self.data_path = data_path
        self.num_imgs = num_imgs 
        self.target_imgsize = target_imgsize 

    def normalize_np_image(self, image):
        return (image / 255.0 - 0.5) / 0.5

    def denormalize_np_image(self, image):
        return (image * 0.5 + 0.5) * 255

    def get_input(self, image_path):
        image = np.array(Image.open(image_path)).astype(np.float32)
        return self.normalize_np_image(image)

    def get_imagelist(self, data_path, celebA=False): 
        if celebA == True:
            imgs_path = os.path.join(data_path, 'img_align_celeba/*.jpg')
        else:
            imgs_path = os.path.join(data_path, '*.jpg') 

        all_namelist = glob.glob(imgs_path, recursive=True)
        return all_namelist[:self.num_imgs]

    def load_and_preprocess_image(self, image_path): 
        image = Image.open(image_path)
        j = (image.size[0] - 100) // 2
        i = (image.size[1] - 100) // 2
        image = image.crop([j, i, j + 100, i + 100])    
        image = image.resize([self.target_imgsize, self.target_imgsize], Image.BILINEAR)
        image = np.array(image.convert('RGB')).astype(np.float32)
        image = self.normalize_np_image(image)
        return image    

    #reads data, preprocesses and saves to another folder with the given path. 
    def preprocess_and_save_images(self, dir_name, save_path=''): 
        preproc_folder_path = os.path.join(save_path, dir_name)
        if not os.path.exists(preproc_folder_path):
            os.makedirs(preproc_folder_path)   
            imgs_path = os.path.join(self.data_path, 'img_align_celeba/*.jpg')
            print('Saving and preprocessing images ...')
            for num, imgname in enumerate(glob.iglob(imgs_path, recursive=True)):
                cur_image = self.load_and_preprocess_image(imgname)
                cur_image = Image.fromarray(np.uint8(self.denormalize_np_image(cur_image)))
                cur_image.save(preproc_folder_path + '/preprocessed_image_%d.jpg' %(num)) 
        self.data_path= preproc_folder_path

    def get_nextbatch(self, batch_size): 
        assert (batch_size > 0),"Give a valid batch size"
        cur_idx = 0
        image_namelist = self.get_imagelist(self.data_path)
        while cur_idx + batch_size <= self.num_imgs:
            cur_namelist = image_namelist[cur_idx:cur_idx + batch_size]
            cur_batch = [self.get_input(image_path) for image_path in cur_namelist]
            cur_batch = np.array(cur_batch).astype(np.float32)
            cur_idx += batch_size
            yield cur_batch

    def show_image(self, image, normalized=True):
        if not type(image).__module__ == np.__name__:
            image = image.numpy()
        if normalized:
            npimg = (image * 0.5) + 0.5 
        npimg.astype(np.uint8)
        plt.imshow(npimg, interpolation='nearest')



def generator(x, args, reuse=False):
    with tf.device('/gpu:0'):
        with tf.variable_scope("generator", reuse=reuse): 
            #Layer Block 1
            with tf.variable_scope("layer1"):
                deconv1 = tf.layers.conv2d_transpose(inputs=x, 
                                             filters= args.n_g_filters*8, 
                                             kernel_size=4, 
                                             strides=1,
                                             padding='valid',
                                             use_bias=False,
                                             name='deconv')
                batch_norm1=tf.layers.batch_normalization(deconv1,
                                             name = 'batch_norm')
                relu1 = tf.nn.relu(batch_norm1, name='relu')
            #Layer Block 2
            with tf.variable_scope("layer2"):
                deconv2 = tf.layers.conv2d_transpose(inputs=relu1, 
                                             filters=args.n_g_filters*4, 
                                             kernel_size=4,
                                             strides=2,
                                             padding='same', 
                                             use_bias=False,
                                             name='deconv')
                batch_norm2 = tf.layers.batch_normalization(deconv2,
                                             name = 'batch_norm')
                relu2 = tf.nn.relu(batch_norm2, name='relu')
            #Layer Block 3
            with tf.variable_scope("layer3"):
                deconv3 = tf.layers.conv2d_transpose(inputs=relu2, 
                                             filters=args.n_g_filters*2, 
                                             kernel_size=4, 
                                             strides=2, 
                                             padding='same',
                                             use_bias = False,
                                             name='deconv')
                batch_norm3 = tf.layers.batch_normalization(deconv3, 
                                             name = 'batch_norm')
                relu3 = tf.nn.relu(batch_norm3, name='relu')
            #Layer Block 4
            with tf.variable_scope("layer4"):
                deconv4 = tf.layers.conv2d_transpose(inputs=relu3, 
                                             filters=args.n_g_filters, 
                                             kernel_size=4, 
                                             strides=2,
                                             padding='same',
                                             use_bias=False,
                                             name='deconv')
                batch_norm4 = tf.layers.batch_normalization(deconv4,
                                             name = 'batch_norm')
                relu4 = tf.nn.relu(batch_norm4, name='relu')
            #Output Layer
            with tf.variable_scope("last_layer"):
                logit = tf.layers.conv2d_transpose(inputs=relu4, 
                                             filters=3, 
                                             kernel_size=4, 
                                             strides=2, 
                                             padding='same',
                                             use_bias=False,
                                             name='logit')
                output = tf.nn.tanh(logit) 
    return output, logit



def discriminator(x, args, reuse=False):
    with tf.device('/gpu:0'):
        with tf.variable_scope("discriminator", reuse=reuse): 
            with tf.variable_scope("layer1"):
                conv1 = tf.layers.conv2d(inputs=x,
                                         filters=args.n_f_filters,
                                         kernel_size=4,
                                         strides=2,
                                         padding='same',
                                         use_bias=False,
                                         name='conv')
                relu1 = tf.nn.leaky_relu(conv1, alpha=0.2, name='relu')
            with tf.variable_scope("layer2"):
                conv2 = tf.layers.conv2d(inputs=relu1,
                                         filters=args.n_f_filters*2,
                                         kernel_size=4,
                                         strides=2,
                                         padding='same',
                                         use_bias=False,
                                         name='conv')
                batch_norm2 = tf.layers.batch_normalization(conv2,name='batch_norm')
                relu2 = tf.nn.leaky_relu(batch_norm2, alpha=0.2, name='relu')
            with tf.variable_scope("layer3"):
                conv3 = tf.layers.conv2d(inputs=relu2,
                                         filters=args.n_f_filters*4,
                                         kernel_size=4,
                                         strides=2,
                                         padding='same',
                                         use_bias=False,
                                         name='conv')
                batch_norm3 = tf.layers.batch_normalization(conv3, name='batch_norm')
                relu3 = tf.nn.leaky_relu(batch_norm3, name='relu')
            with tf.variable_scope("layer4"):
                conv4 = tf.layers.conv2d(inputs=relu3,
                                         filters=args.n_f_filters*8,
                                         kernel_size=4,
                                         strides=2,
                                         padding='same',
                                         use_bias=False,
                                         name='conv')
                batch_norm4 = tf.layers.batch_normalization(conv4, name='batch_norm')
                relu4 = tf.nn.leaky_relu(batch_norm4, alpha=0.2, name='relu')
            with tf.variable_scope("last_layer"):
                logit = tf.layers.conv2d(inputs=relu4,
                                         filters=1,
                                         kernel_size=4,
                                         strides=1,
                                         padding='valid',
                                         use_bias=False,
                                         name='conv')
                output = tf.nn.sigmoid(logit) 
    return output, logit



def sample_z(dim_z, num_batch):
    mu = 0
    sigma = 1
    s = np.random.normal(mu, sigma, num_batch*dim_z)
    samples = s.reshape(num_batch, 1, 1, dim_z)
    ##dist = tf.distributions.Normal(0.0, 1.0)
    ##samples = dist.sample([num_batch, 1, 1, dim_z])
    return samples
#64,1,1,100  6400
sample_z(100, 64)



def get_losses(d_real_logits, d_fake_logits):
    #add new loss function here  
    ###d_loss_real = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=d_real_logits, labels=tf.ones_like(d_real_logits)))    
    ###d_loss_fake = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=d_fake_logits, labels=tf.zeros_like(d_fake_logits))) 
    ###d_loss = d_loss_real + d_loss_fake
    ###g_loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=d_fake_logits, labels=tf.ones_like(d_fake_logits)))
    ###return d_loss, g_loss
    d_loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=d_real_logits,labels=tf.ones_like(d_real_logits)) + tf.nn.sigmoid_cross_entropy_with_logits(logits=d_fake_logits,labels=tf.zeros_like(d_fake_logits)))

    g_loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=d_fake_logits,labels=tf.ones_like(d_fake_logits)))
    return d_loss, g_loss



def get_optimizers(learning_rate, beta1, beta2):
    d_optimizer = tf.train.AdamOptimizer(learning_rate, beta1, beta2)
    g_optimizer = tf.train.AdamOptimizer(learning_rate, beta1, beta2)
    return d_optimizer, g_optimizer


def optimize(d_optimizer, g_optimizer, d_loss, g_loss):
    d_step = d_optimizer.minimize(d_loss)
    g_step = g_optimizer.minimize(g_loss)
    return d_step, g_step


LOGDIR = "logs_basic_dcgan"

def merge_images(image_batch, size):
    h,w = image_batch.shape[1], image_batch.shape[2]
    c = image_batch.shape[3]
    img = np.zeros((int(h*size[0]), w*size[1], c))
    for idx, im in enumerate(image_batch):
        i = idx % size[1]
        j = idx // size[1]
        img[j*h:j*h+h, i*w:i*w+w,:] = im
    return img
itr_fh = open('basic_gan_itr.txt', 'a+')

def train(args):
    tf.reset_default_graph()
    data_loader = Dataset(args.data_path, args.num_images, args.image_size) 
    #data_loader.preprocess_and_save_images('preprocessed', 'results_celebA') #preprocess the images once
    X = tf.placeholder(tf.float32, shape=[args.batch_size, args.image_size , args.image_size, 3])
    Z = tf.placeholder(tf.float32, shape=[args.batch_size, 1, 1, args.dim_z])

    G_sample, _ = generator(Z, args)
    D_real, D_real_logits = discriminator(X, args)
    D_fake, D_fake_logits = discriminator(G_sample, args, reuse=True)
    d_loss, g_loss = get_losses(D_real_logits, D_fake_logits)
    d_optimizer, g_optimizer = get_optimizers(args.lr, args.beta1, args.beta2)
    d_step, g_step = optimize(d_optimizer, g_optimizer, d_loss, g_loss)
    ###z_sum = tf.summary.histogram('z', Z)
    ###d_sum = tf.summary.histogram('d', D_real)
    ###G_sum = tf.summary.histogram('g', G_sample)
    ###d_loss_sum = tf.summary.scalar('d_loss', d_loss)
    ###g_loss_sum = tf.summary.scalar('g_loss', g_loss)
    ###d_sum = tf.summary.merge([z_sum, d_sum, d_loss_sum])
    ###g_sum = tf.summary.merge([z_sum, G_sum, g_loss_sum])
    ###saver = tf.train.Saver()
    ###merged_summary = tf.summary.merge_all()

    ###d_loss_summary = tf.summary.scalar("Discriminator_Total_Loss", d_loss)
    ###g_loss_summary = tf.summary.scalar("Generator_Total_Loss", g_loss)
    ###merged_summary = tf.summary.merge_all()

    with tf.Session() as sess:

        sess.run(tf.global_variables_initializer())
        for epoch in range(args.n_epoch):
            for itr, real_batch in enumerate(data_loader.get_nextbatch(args.batch_size)):
                print('itr is %d, and epoch is %d' %(itr, epoch))
                itr_fh.write("epoch: " +  str(epoch) + " itr: " + str(itr) + "\n")

                Z_sample = sample_z(args.dim_z, args.batch_size)

                _, _ = sess.run([d_step, g_step], feed_dict={X:real_batch , Z:Z_sample})
                sample = sess.run(G_sample, feed_dict={Z:Z_sample})
                print("sample size is: ", sample.shape)
                if itr==3164: #num_images/batch_size
                    im_merged = merge_images(sample[:16], [4,4])
                    plt.imsave('sample_gan_images/im_merged_epoch_%d.png' %(epoch), im_merged )
                    scipy.misc.imsave('sample_gan_images/im_epoch_%d_itr_%d.png' %(epoch,itr), sample[1])
                    ##merged_summary = sess.run(merged_summary, feed_dict={X:real_batch , Z:Z_sample})
                    ###writer = tf.summary.FileWriter(LOGDIR)   
                    ###writer.add_summary(merged_summary, itr)
                    ###d_loss_summary = tf.summary.scalar("Discriminator_Total_Loss", d_loss)
                    ###g_loss_summary = tf.summary.scalar("Generator_Total_Loss", g_loss)
                    ###merged_summary = tf.summary.merge_all()
                    ###writer.add_graph(sess.graph)
                    ###saver.save(sess, save_path='logs_basic_dcgan/gan.ckpt')


train(args)

Here is the images created at the end of first 5 epochs. I also have commented stuff related to tensorboard because it makes it very slow unfortunately.

end of Epoch 0:

end of Epoch 1:

end of Epoch 2:

end of Epoch 3:

end of Epoch 4:

Generative Adversarial Networks (GANs)

Can the input of a Generative Adversarial Network (GAN) be an image instead of a random distribution?
forcing a column to be primary split column in RandomForest

I have data for various institutes such that certain institutes provide us more fields than others. These additional data fields seem to have a high correlation to the binary outcome we are trying to predict, so ignoring them is not an option. Also, we don't want build institute specific models.

One of the options we are considering is including institute value as a feature with that idea that a single model will consider it to be the feature used for splitting primarily. Thus, if we imagine a tree based model, each institute gets it's own tree in a single model.

How could we force a feature to be the primary split feature?
Probabilistic classification with Gaussian Bayes Classifier vs Logistic Regression
I have a binary classification problem where I have a few great features that have the power to predict almost 100% of the test data because the problem is relatively simple.

However, as the nature of the problem requires, I have no luxury to make mistake(let's say) so instead of giving a prediction I am not sure of, I would rather have the output as probability, set a threshold and would be able to say, "if I am less than %95 sure, I will call this "NOT SURE" and act accordingly". Saying "I don't know" rather than making a mistake is better.

So far so good.

For this purpose, I tried Gaussian Bayes Classifier(I have a cont. feature) and Logistic Regression algorithms, which provide me the probability as well as the prediction for the classification.

Coming to my Problem:
- GBC has around 99% success rate while Logistic Regression has lower, around 96% success rate. So I naturally would prefer to use GBC. However, as successful as GBC is, it is also very sure of itself. The odds I get are either 1 or very very close to 1, such as 0.9999997, which makes things tough for me, because in practice GBC does not provide me probabilities now.
- Logistic Regression works poor, but at least gives better and more 'sensible' odds.
As nature of my problem, the cost of misclassifying is by the power of 2 so if I misclassify 4 of the products, I lose 2^4 more (it's unit-less but gives an idea anyway).

In the end; I would like to be able to classify with a higher success than Logistic Regression, but also be able to have more probabilities so I can set a threshold and point out the ones I am not sure of.

Any suggestions?

Thanks in advance.

Python: Imbalanced data for XGBoost Multi-label classification

I have a dataset of a stock's returns where the Y-label is price change direction (= 2 if upward tick, = 1 if downward tick, and = 0 if no move. Some of the features, X, include the lagged label values (i.e. the previous day's price direction change).

I am trying to run the XGBoost classification model, however my data is highly imbalanced. Most of the Y label values are = 0 meaning the stock price did not move.

How can I incorporate this imbalance in a multi-label XGBoost classification problem?

My code is the following:

X = df[["ret_D_lag_1", "ret_D_lag_2", "ret_D_lag_3"]]
y = df["ret_D_t1"]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=123)

# use DMatrix for xgboost
dtrain = xgb.DMatrix(X_train, label=y_train)
dtest = xgb.DMatrix(X_test, label=y_test)

# set xgboost params
param = {
    'max_depth': 3,  # the maximum depth of each tree
    'eta': 0.3,  # the training step for each iteration
    'silent': 1,  # logging mode - quiet
    'objective': 'multi:softprob',  # error evaluation for multiclass training
    'num_class': 3}  # the number of classes that exist in this datset
num_round = 20  # the number of training iterations

# Train the model
bst = xgb.train(param, dtrain, num_round)

# Predict and choose highest probability for each label
preds = bst.predict(dtest)
best_preds = np.asarray([np.argmax(line) for line in preds])

What is wrong with my implementation of AdaBoost?

I tried to implement the AdaBoost algorithm of Freund and Schapire as close to the original as possible (see p. 2 here: http://rob.schapire.net/papers/explaining-adaboost.pdf):

library(rpart)
library(OneR)

maxdepth <- 1
T <- 100 # number of rounds

# Given: (x_1, y_1),...,(x_m, y_m) where x_i element of X, y_i element of {-1, +1}
myocarde <- read.table("http://freakonometrics.free.fr/myocarde.csv", head = TRUE, sep = ";")
#myocarde <- read.table("data/myocarde.csv", header = TRUE, sep = ";")
y <- (myocarde[ , "PRONO"] == "SURVIE") * 2 - 1
x <- myocarde[ , 1:7]
m <- nrow(x)
data <- data.frame(x, y)

# Initialize: D_1(i) = 1/m for i = 1,...,m
D <- rep(1/m, m)

H <- replicate(T, list())
a <- vector(mode = "numeric", T)
set.seed(123)

# For t = 1,...,T
for(t in 1:T) {
  # Train weak learner using distribution D_t
  # Get weak hypothesis h_t: X -> {-1, +1}
  data_D_t <- data[sample(m, 10*m, replace = TRUE, prob = D), ]
  H[[t]] <- rpart(y ~., data = data_D_t, maxdepth = maxdepth, method = "class") # error: number of target levels must be bigger than 1
  # Aim: select h_t with low weighted error: e_t = Pr_i~D_t[h_t(x_i) != y_i]
  h <- predict(H[[t]], x, type = "class")
  e <- sum(h != y) / m
  # Choose a_t = 0.5 * log((1-e) / e)
  a[t] <- 0.5 * log((1-e) / e)
  # Update for i = 1,...,m: D_t+1(i) = (D_t(i) * exp(-a_t * y_i * h_t(x_i))) / Z_t
  # where Z_t is a normalization factor (chosen so that Dt+1 will be a distribution) 
  D <- D * exp(-a[t] * y * as.numeric(h))
  D <- D / sum(D)
}
# Output the final hypothesis: H(x) = sign(sum of a_t * h_t(x) for t=1 to T)
newdata <- x
H_x <- sapply(H, function(x) as.numeric(as.character(predict(x, newdata = newdata, type = "class"))))
H_x <- t(a * t(H_x))
pred <- sign(rowSums(H_x))

#H
#a
eval_model(pred, y)
## 
## Confusion matrix (absolute):
##           Actual
## Prediction -1  1 Sum
##        -1   0  1   1
##        1   29 41  70
##        Sum 29 42  71
## 
## Confusion matrix (relative):
##           Actual
## Prediction   -1    1  Sum
##        -1  0.00 0.01 0.01
##        1   0.41 0.58 0.99
##        Sum 0.41 0.59 1.00
## 
## Accuracy:
## 0.5775 (41/71)
## 
## Error rate:
## 0.4225 (30/71)
## 
## Error rate reduction (vs. base rate):
## -0.0345 (p-value = 0.6436)

As can be seen the accuracy of the model is horrible compared to other AdaBoost implementations, e.g.:

library(JOUSBoost)
## JOUSBoost 2.1.0
boost <- adaboost(as.matrix(x), y, tree_depth = maxdepth, n_rounds = T)
pred <- predict(boost, x)
eval_model(pred, y)
## 
## Confusion matrix (absolute):
##           Actual
## Prediction -1  1 Sum
##        -1  29  0  29
##        1    0 42  42
##        Sum 29 42  71
## 
## Confusion matrix (relative):
##           Actual
## Prediction   -1    1  Sum
##        -1  0.41 0.00 0.41
##        1   0.00 0.59 0.59
##        Sum 0.41 0.59 1.00
## 
## Accuracy:
## 1 (71/71)
## 
## Error rate:
## 0 (0/71)
## 
## Error rate reduction (vs. base rate):
## 1 (p-value < 2.2e-16)

My question
Could you please give me a hint what went wrong in my implementation? Thank you

Is boosting sensitive to outliers? Why?

1) Are this methods sensitive to outliers?

2) At what level are they? I mean, this sensitivity is due to trees or is something strictly related with boosting?

3) What part of Boosting methods are sensitive to outliers? Imagine that trees are used. It regards the fact that an outlier impacts on the choise of the threshold for binary splitting? Or there is something else?

Mathematics is welcome!
Select attribute in weka ends up 0 for all attributes

I have a data set with 46 attributes and 74170 instances in weka. I suppose I need to select best attributes for better accuracy. first I tried to select attributes using filter with CfsSubsetEval and BestFirst for search, So weka decreased my attributes from 47 to 17 number. Then I used Select Attributes tab for testing my data accuracy and error rate, I used WrapperSubsetEval for evaluator and Adaboost for classifier and naive Bayes for base learner and BestFirst for search and 10 fold cross validation for training. but at the end all the attributes get '0'. I am not sure what is wrong.

Related questions

How to apply the learned model using boosting technique

See also questions close to this topic