Difference between tf.train.shuffle_batch_join and tf.train.shuffle_batch
Looking at both function signatures, with arguments
tf.train.shuffle_batch_join(
tensors_list,
batch_size,
capacity,
min_after_dequeue,
seed=None,
enqueue_many=False,
shapes=None,
allow_smaller_final_batch=False,
shared_name=None,
name=None
)
and
tf.train.shuffle_batch(
tensors,
batch_size,
capacity,
min_after_dequeue,
num_threads=1,
seed=None,
enqueue_many=False,
shapes=None,
allow_smaller_final_batch=False,
shared_name=None,
name=None
)
the only difference is among the atguments is num_threads that denotes intuitively that tf.train.shuffle_batch could be processed with multiple threads or processes, Except that, they seem to do pretty much the same work.
I was wondering if there is a fundamental difference on which someone might choose one over the other except multiprocessing of batches.
1 answer
-
answered 2018-07-11 13:48
Olivier Dehaene
Quoting from the shuffle_batch_join TF documentation :
The tensors_list argument is a list of tuples of tensors, or a list of dictionaries of tensors. Each element in the list is treated similarly to the tensors argument of tf.train.shuffle_batch().
Basically, shuffle_batch_join expects to:
- Receive a list of tensors
- Perform shuffle_batch on each member of the list
- Return a list of tensors with the same number and types as tensors_list[i].
Be aware that if you use shuffle_batch_join :
len(tensors_list) threads will be started, with thread i enqueuing the tensors from tensors_list[i]. tensors_list[i1][j] must match tensors_list[i2][j] in type and shape, except in the first dimension if enqueue_many is true.
See also questions close to this topic
-
How to use the deep-deep project?
Enlace del proyecto:
https://github.com/TeamHG-Memex/deep-deep
I have created a .SH file to install all its dependencies:
#!/bin/bash echo "Instalando proyecto deep-deep IA" apt-get install build-essential checkinstall apt-get install libreadline-gplv2-dev libncursesw5-dev libssl-dev libsqlite3-dev tk-dev libgdbm-dev libc6-dev libbz2-dev cd /usr/src wget https://www.python.org/ftp/python/3.5.5/Python-3.5.5.tgz tar xzf Python-3.5.5.tgz cd Python-3.5.5 sudo ./configure --enable-optimizations sudo make altinstall apt-get install python3-pip pip3 install pytest apt-get install pytest-twisted pip install pytest-twisted python3 -m pip install -U git+git://github.com/python/mypy.git apt install tox git clone https://github.com/TeamHG-Memex/deep-deep.git cd deep-deep/ toxafter installation I have installed run examples / standalone.py:
python3 standalone.py install /usr/local/lib/python3.6/dist-packages/scikit_learn-0.20.0-py3.6-linux-x86_64.egg/sklearn/externals/joblib/externals/cloudpickle/cloudpickle.py:47: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import impI also ran setup.py, How can I make this project work?
-
Performance and lightness of an application: import the whole library or just the necessary modules?
I've been asking myself questions for some time now. If I have to distribute an application that uses large libraries, or frameworks, can I afford to import the entire library or framework, and call the modules when I need them, for example:
import os import nltk import codecs import json import collections import flask #e.g. json.load(json_data) collection.Counter(aList) nltk.word_tokenize(sentence) codecs.open(aFile) os.listdir(aPath)or otherwise it is better to import only the necessary modules, in order to promote lightness and performance, for example:
from collections import Counter from flask import request, Flask, render_template, redirect, url_for from nltk.data import load from nltk import word_tokenize from nltk.corpus import stopwords.words as stopW from os import listdirIt makes me very curious to know which is the best solution when we aim to distribute an application on any platform (whether desktop, mobile, or web based). Thank you very much in advance.
-
How can I send a zip file on the disk to the client with flask and ajax?
I saw this question, but I can't use that answer since the ZipFile is generated in another script. In the flask app, I access that zip file saved on the disk.
if download_local: file_name = "adyenfiles.zip" file_path = os.path.join(app.config["tmp_path"], file_name).replace('\\', '/') logging.info(f"Enviando arquivo: {file_path}") logging.info(f"FILE_PATH: {file_path}") response = make_response(send_file(file_path, as_attachment=True, mimetype="application/zip")) response.headers["Content-Disposition"] = f"attachment;filename={file_name}" return responseThen on the client:
$.ajax({ type: "POST", url: url, data: $("#manual-download-form").serialize(), beforeSend: function () { $("#request-download-progress").show(); displayMessage("Processo de requisição de download iniciado, por favor aguarde."); }, success: function (response, status, xhr) { $("#manual-download-form")[0].reset(); $("input[name='date-group']:checked").val("specific"); $("#date-input-area").html( ` <div class="input-field col s12"> <input id="manual-download-datepicker" name="specific-date" type="text" class="datepicker"> <label for="specific-date">Data*</label> </div> ` ); resetDatepicker(); var filename = ""; var disposition = xhr.getResponseHeader('Content-Disposition'); if (disposition && disposition.indexOf('attachment') !== -1) { var filenameRegex = /filename[^;=\n]*=((['"]).*?\2|[^;\n]*)/; var matches = filenameRegex.exec(disposition); if (matches != null && matches[1]) filename = matches[1].replace(/['"]/g, ''); } var type = xhr.getResponseHeader('Content-Type'); var blob = new Blob([response], { type: "application/zip" }); if (typeof window.navigator.msSaveBlob !== 'undefined') { // IE workaround for "HTML7007: One or more blob URLs were revoked by closing the blob for which they were created. These URLs will no longer resolve as the data backing the URL has been freed." window.navigator.msSaveBlob(blob, filename); } else { var URL = window.URL || window.webkitURL; var downloadUrl = URL.createObjectURL(blob); if (filename) { // use HTML5 a[download] attribute to specify filename var a = document.createElement("a"); // safari doesn't support this yet if (typeof a.download === 'undefined') { window.location = downloadUrl; } else { a.href = downloadUrl; a.download = filename; document.body.appendChild(a); a.click(); } } else { window.location = downloadUrl; } setTimeout(function () { URL.revokeObjectURL(downloadUrl); }, 100); // cleanup } }, complete: function (response, status, xhr) { $("#request-download-progress").hide(); displayMessage("Se necessário, verifique o diretório para certificar-se de que os arquivos foram baixados."); }, error: function (response) { displayMessage(response["responseText"]); } });I get this error when I try to open the zip file (Stating that it can't be open as a zip file):
How can I send a zip file that is saved on the disk to the client?? How do I set it up on flask and generate the blob on the client?
-
Python C-API… how to write python code in C
I have the following python code… and want to write this Code in… Python-C-API
from enum import Enum class MqWaitOnEventE(Enum): NO = 0 ONCE = 1 FOREVER = 2Thanks for help.
-
disable Tensorflow eager execution
I am trying to learn TF and working with Placeholders. When I try create the Placeholder I get an error:
RuntimeError: tf.placeholder() is not compatible with eager executionwhich makes sense as Placeholders are not executable immediately.My struggle is in finding how to turn eager execution off?
I have never turned eager execution on in the first place so not sure how it happened. Is there an opposite to
tf.disable_eager_execution()?Thank you
-
ML Engine Runtime version and Python version not supported
I am trying to run Google clound ml-engine following this tutorial, when executing this command :
$ gcloud ml-engine jobs submit training `whoami`_object_detection_`date +%s` \ --job-dir=gs://${YOUR_GCS_BUCKET}/train\ --packages dist/object_detection-0.1.tar.gz,slim/dist/slim-0.1.tar.gz,/tmp/pycocotools/pycocotools-2.0.tar.gz\ --module-name object_detection.model_tpu_main \ --runtime-version 1.10\ --scale-tier BASIC_TPU\ --region us-central1\ -- --model_dir=gs://${YOUR_GCS_BUCKET}/train --tpu_zone us-central1 --pipeline_config_path=gs://${YOUR_GCS_BUCKET}/data/pipeline.configcommands which answers me :
ERROR: (gcloud.ml-engine.jobs.submit.training) INVALID_ARGUMENT: Field: runtime_version Error: The specified runtime version '1.10' with the Python version '' is not supported for TPU training. Please specify a different runtime version. See https://cloud.google.com/ml/docs/concepts/runtime-version-list for a list of supported versions - '@type': type.googleapis.com/google.rpc.BadRequest fieldViolations: - description: The specified runtime version '1.10' with the Python version '' is not supported for TPU training. Please specify a different runtime version. See https://cloud.google.com/ml/docs/concepts/runtime-version-list for a list of supported versions field: runtime_versionBecause no version of python is detected, I added a
--config=config.yamlargument to the command line:config.yaml:trainingInput: pythonVersion: "3.5"but the problem barely changed :
... - description: The specified runtime version '1.10' with the Python version '3.5' ...the runtime version list specify that runtime-version 1.10 is compatible with python 3.5, I also tried different runtime environement / python version which are required to work but my command keeps failling.
-
How to update tensorflow without removing the existing one?
I have tensorflow 1.5.0, but i want to update it to the latest version
without removing the existing one. -
Why my handmade numpy neural network doesn't learn?
As an exercise I was building a neural network in numpy from scratch.
For simplicity I wanted to use it to solve XOR problem. I derived all the equation and put everything together, but it looks like my network doesn't learn. I've spent some time trying to spot the mistake, but without success. Maybe you notice something I'm missing here?
X = [(0,0), (1,0), (0,1), (1,1)] Y = [0, 1, 1, 0] w1 = 2 * np.random.random(size=(2,3)) - 1 w2 = 2 * np.random.random(size=(3,1)) - 1 b1 = 2 * np.random.random(size=(1,3)) - 1 b2 = 2 * np.random.random(size=(1,1)) - 1 def sigmoid(x): return 1./(1 + np.exp(-x)) def dsigmoid(y): return y*(1-y) N = 1000 error = np.zeros((N,1)) for n in range(N): Dw_1 = np.zeros((2,3)) Dw_2 = np.zeros((3,1)) Db_1 = np.zeros((1,3)) Db_2 = np.zeros((1,1)) for i in range(len(X)): # iterate over all examples x = np.array(X[i]) y = np.array(Y[i]) # Forward pass, 1st layer act1 = np.dot(w1.T, x) + b1 lay1 = sigmoid(act1) # Forward pass, 2nd layer act2 = np.dot(w2.T, lay1.T) + b2 lay2 = sigmoid(act2) # Computing error E = 0.5*(lay2 - y)**2 error[n] += E[0] # Backprop, 2nd layer delta_l2 = (y-lay2) * dsigmoid(lay2) corr_w2 = (delta_l2 * lay1).T corr_b2 = delta_l2 * 1 # Backprop, 1st layer delta_l1 = np.dot(w2, delta_l2) * dsigmoid(lay1).T corr_w1 = np.outer(x, delta_l1) corr_b1 = (delta_l1 * 1).T Dw_2 += corr_w2 Dw_1 += corr_w1 Db_2 += corr_b2 Db_1 += corr_b1 if n % 1000 == 0: print y, lay2, if n % 1000 == 0: print w2 = w2 - eta * Dw_2 b2 = b2 - eta * Db_2 w1 = w1 - eta * Dw_1 b1 = b1 - eta * Db_1 error[n] /= len(X)
-
Stacking model using iris dataset
In the below code i implemented a stacking model using KNN, SVM and RF as base models with an example on iris dataset the difficulty i faced is that once i use a metamodel to stack the results such as SVM or RF or another models they gives the same results which is weird and abnormal. May anyone advise.
from sklearn import datasets cancer=datasets.load_breast_cancer() X=cancer.data y=cancer.target from sklearn import datasets from sklearn.model_selection import train_test_split from sklearn.model_selection import GridSearchCV from sklearn.metrics import classification_report from sklearn.svm import SVC from sklearn.metrics import accuracy_score from sklearn.model_selection import * from time import * from sklearn import metrics np.random.seed(123) # Split the dataset in two equal parts X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.3, random_state=None) # Set the parameters by cross-validation tuned_parameters = [{'kernel': ['rbf'], 'gamma': [1e-2, 1e-3, 1e-4, 1e-5], 'C': [0.001, 0.10, 0.1, 10, 25, 50, 100, 1000]}, {'kernel': ['sigmoid'], 'gamma': [1e-2, 1e-3, 1e-4, 1e-5], 'C': [0.001, 0.10, 0.1, 10, 25, 50, 100, 1000] },{'kernel': ['linear'], 'C': [0.001, 0.10, 0.1, 10, 25, 50, 100, 1000]}] print() clf = GridSearchCV(SVC(), tuned_parameters, cv=10, scoring='accuracy') t0 = time() clf.fit(X_train, y_train) t = time() - t0 print("Best parameters set found on development set:") print() print(clf.best_params_) print() print('Training accuracy') print(clf.best_score_) print(clf.best_estimator_) print() print() print('****Results****') svm_pred=cross_val_predict(clf.best_estimator_,X_train,y_train,cv=10) #print("\t\taccuracytrainkfold: {}".format(metrics.accuracy_score(y_train, svm_pred))) print("=" * 52) print("time cost: {}".format(t)) #print(clf.cv_results_) print() #print(cross_val_score(clf,X_train,y_train,cv=10).mean()) print("\t\taccuracy: {}".format(metrics.accuracy_score(y_train, svm_pred))) Best parameters set found on development set: {'C': 50, 'gamma': 0.01, 'kernel': 'rbf'} Training accuracy 0.8165938864628821 SVC(C=50, cache_size=200, class_weight=None, coef0=0.0, decision_function_shape='ovr', degree=3, gamma=0.01, kernel='rbf', max_iter=-1, probability=False, random_state=None, shrinking=True, tol=0.001, verbose=False) ****Results**** ==================================================== time cost: 8.607818841934204 accuracy: 0.8165938864628821 In [28]: opt_svm_pred=clf.best_estimator_.predict(X_test) print() print("confusion matrix\n", metrics.confusion_matrix(y_test, opt_svm_pred)) print() print("\t\taccuracy: {}".format(metrics.accuracy_score(y_test, opt_svm_pred))) print("\t\troc_auc_score: {}".format(metrics.roc_auc_score(y_test, opt_svm_pred))) print("\t\tcohen_kappa_score: {}".format(metrics.cohen_kappa_score(y_test, opt_svm_pred))) print() print("\t\tclassification report") print("-" * 52) print(metrics.classification_report(y_test, opt_svm_pred)) confusion matrix [[45 8] [ 6 40]] accuracy: 0.8585858585858586 roc_auc_score: 0.8593109105824446 cohen_kappa_score: 0.7165644171779141 classification report ---------------------------------------------------- precision recall f1-score support 0 0.88 0.85 0.87 53 1 0.83 0.87 0.85 46 avg / total 0.86 0.86 0.86 99 RF¶ In [19]: from sklearn.model_selection import * from sklearn.ensemble import RandomForestClassifier #making the instance np.random.seed(123) # Split the dataset in two equal parts X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.3, random_state=None) model=RandomForestClassifier() #hyper parameters set params = {'criterion':['gini','entropy'], 'n_estimators':[10,15,20,25,30], 'min_samples_leaf':[1,2,3], 'min_samples_split':[3,4,5,6,7], 'random_state':[123], 'n_jobs':[-1]} #Making models with hyper parameters sets RF_grid = GridSearchCV(model, param_grid=params,cv=10, n_jobs=-1) #learning t0 = time() RF_grid.fit(X_train,y_train) t0 = time()-t0 #The best hyper parameters set print("Best Hyper Parameters:\n",RF_grid.best_params_) print("training accuracy:\n",RF_grid.best_score_) #Prediction RF_prediction=cross_val_predict(RF_grid.best_estimator_,X_train,y_train,cv=10) #print("\t\taccuracytrainkfold: {}".format(metrics.accuracy_score(y_train, svm_pred))) print("=" * 52) print("time cost: {}".format(t)) #print(clf.cv_results_) print() #print(cross_val_score(clf,X_train,y_train,cv=10).mean()) print("\t\taccuracy: {}".format(metrics.accuracy_score(y_train, RF_prediction))) Best Hyper Parameters: {'criterion': 'gini', 'min_samples_leaf': 1, 'min_samples_split': 5, 'n_estimators': 30, 'n_jobs': -1, 'random_state': 123} training accuracy: 0.8558951965065502 ==================================================== time cost: 326.25868034362793 accuracy: 0.8558951965065502 In [25]: opt_RF_pred=RF_grid.best_estimator_.predict(X_test) #evaluation(Confusion Matrix) #importing the metrics module from sklearn import metrics #evaluation(Accuracy) print("Accuracy:",metrics.accuracy_score(opt_RF_pred,y_test)) #evaluation(Confusion Matrix) print("Confusion Matrix:\n",metrics.confusion_matrix(opt_RF_pred,y_test)) print("\t\tclassification report") print("-" * 52) print(metrics.classification_report(opt_RF_pred,y_test)) Accuracy: 0.8383838383838383 Confusion Matrix: [[40 3] [13 43]] classification report ---------------------------------------------------- precision recall f1-score support 0 0.75 0.93 0.83 43 1 0.93 0.77 0.84 56 avg / total 0.86 0.84 0.84 99 KNN¶ In [21]: from sklearn.model_selection import GridSearchCV from sklearn.neighbors import KNeighborsClassifier import matplotlib.pyplot as plt from sklearn.model_selection import * from sklearn import metrics from time import * np.random.seed(123) # split into train and test X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=None) # instantiate model knn = KNeighborsClassifier() # create a parameter grid: map the parameter names to the values that should be searched # simply a python dictionary # key: parameter name # value: list of values that should be searched for that parameter # single key-value pair for param_grid param_grid = {'n_neighbors':[1,2,3,4,5,6,7,8,9,10],'leaf_size':[1,2,3,5], 'weights':['uniform', 'distance'], 'algorithm':['auto'], 'n_jobs':[-1]} # instantiate the grid grid_knn = GridSearchCV(knn, param_grid, cv=10, scoring='accuracy') # fit the grid with data t0=time() grid_knn.fit(X_train, y_train) t = time() - t0 # view the complete results (list of named tuples) #The best hyper parameters set print("Best Hyper Parameters:\n",grid_knn.best_params_) print("training accuracy:\n",grid_knn.best_score_) #Prediction knn_prediction=cross_val_predict(grid_knn.best_estimator_,X_train,y_train,cv=10) print("=" * 52) print("time cost: {}".format(t)) print() #{'algorithm': 'auto', 'leaf_size': 1, 'n_jobs': -1, 'n_neighbors': 8, 'weights': 'uniform'} #print("\t\taccuracytrainkfold: {}".format(metrics.accuracy_score(y_train, svm_pred))) print("=" * 52) print("time cost: {}".format(t)) #print(clf.cv_results_) print() #print(cross_val_score(clf,X_train,y_train,cv=10).mean()) print("\t\taccuracy: {}".format(metrics.accuracy_score(y_train, knn_prediction))) Best Hyper Parameters: {'algorithm': 'auto', 'leaf_size': 1, 'n_jobs': -1, 'n_neighbors': 3, 'weights': 'uniform'} training accuracy: 0.7991266375545851 ==================================================== time cost: 186.73932242393494 ==================================================== time cost: 186.73932242393494 accuracy: 0.7991266375545851 In [24]: opt_knn_pred=grid_knn.best_estimator_.predict(X_test) #importing the metrics module from sklearn import metrics #evaluation(Accuracy) print("Accuracy:",metrics.accuracy_score(opt_knn_pred,y_test)) #evaluation(Confusion Metrix) print("Confusion Matrix:\n",metrics.confusion_matrix(opt_knn_pred,y_test)) print() print("\t\taccuracy: {}".format(metrics.accuracy_score(opt_knn_pred,y_test))) print(metrics.classification_report(opt_knn_pred,y_test)) Accuracy: 0.8181818181818182 Confusion Matrix: [[41 6] [12 40]] accuracy: 0.8181818181818182 precision recall f1-score support 0 0.77 0.87 0.82 47 1 0.87 0.77 0.82 52 avg / total 0.82 0.82 0.82 99 PNN STACKING¶ In [29]: X_pred=np.array([svm_pred,knn_prediction,RF_prediction]).T #pred=np.array([pred1,pred2,pred3,pred4]).T #output layer containg y_test y_train #split dataset from sklearn.neural_network import MLPClassifier from sklearn.model_selection import train_test_split from sklearn.model_selection import GridSearchCV from time import * from neupy.algorithms import PNN from time import * np.random.seed(123) # Set the parameters by cross-validation tuned_parameters ={'std':[0.05,0.1,0.15,0.2,0.25,0.3,0.35,0.4,0.45,0.5,0.55,0.6,0.65,0.7,0.75,0.8,0.85,0.9,0.95,1] } print() grid_pnnstack = GridSearchCV(PNN(verbose=False), tuned_parameters, cv=10, scoring='accuracy') t0=time() grid_pnnstack.fit(X_pred, y_train) t = time() - t0 print("Best Hyper Parameters:\n",grid_pnnstack.best_params_) print(grid_pnnstack.best_score_) Time: 0:00:00 |N/A%| | ETA: --:--:-- Best Hyper Parameters: {'std': 0.45} 0.851528384279476 In [30]: X_opt_pred=np.array([opt_svm_pred,opt_knn_pred,opt_RF_pred]).T #output layer containg y_test y_test final_prediction=grid_pnnstack.best_estimator_.predict(X_opt_pred) print("=" * 52) print("time cost: {}".format(t)) print() #importing the metrics module from sklearn import metrics #evaluation(Accuracy) print("Accuracy:",metrics.accuracy_score(final_prediction,y_test)) #evaluation(Confusion Metrix) print("Confusion Metrix:\n",metrics.confusion_matrix(final_prediction,y_test)) print() print("\t\taccuracy: {}".format(metrics.accuracy_score(final_prediction, y_test))) print(metrics.classification_report(final_prediction,y_test)) ==================================================== time cost: 3.23099684715271 Accuracy: 0.8383838383838383 Confusion Metrix: [[40 3] [13 43]] accuracy: 0.8383838383838383 precision recall f1-score support 0 0.75 0.93 0.83 43 1 0.93 0.77 0.84 56 avg / total 0.86 0.84 0.84 99 MLP Stacking¶ In [30]: X_pred=np.array([svm_pred,knn_prediction,RF_prediction]).T #pred=np.array([pred1,pred2,pred3,pred4]).T #output layer containg y_test y_train #split dataset from sklearn.neural_network import MLPClassifier from sklearn.model_selection import train_test_split from sklearn.model_selection import GridSearchCV from time import * np.random.seed(123) param_grid = { 'activation' : ['identity', 'logistic', 'tanh', 'relu'], 'solver' :['lbfgs', 'sgd', 'adam'] , 'alpha':[0.0001,1e-5,0.01,0.001], 'batch_size' :[1000,5000], 'learning_rate':['constant', 'invscaling', 'adaptive'] , 'learning_rate_init':[0.001,0.01,0.1,0.2,0.3] , 'hidden_layer_sizes':[[4,4],[4,5],[4,6],[4,7],[4,8],[4,9]] } mlp = MLPClassifier() grid_mlp = GridSearchCV(mlp, param_grid, cv=10, scoring='accuracy',n_jobs=-1) t0=time() grid_mlp.fit(X_pred, y_train) t = time() - t0 print("Best Hyper Parameters:\n",grid_mlp.best_params_) print(grid_mlp.best_score_) Best Hyper Parameters: {'activation': 'identity', 'alpha': 0.01, 'batch_size': 5000, 'hidden_layer_sizes': [4, 8], 'learning_rate': 'constant', 'learning_rate_init': 0.1, 'solver': 'adam'} 0.8646288209606987 C:\Users\ixuser\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:358: UserWarning: Got `batch_size` less than 1 or larger than sample size. It is going to be clipped warnings.warn("Got `batch_size` less than 1 or larger than " In [31]: X_opt_pred=np.array([opt_svm_pred,opt_knn_pred,opt_RF_pred]).T #output layer containg y_test y_test final_prediction=grid_mlp.best_estimator_.predict(X_opt_pred) print("=" * 52) print("time cost: {}".format(t)) print() #importing the metrics module from sklearn import metrics #evaluation(Accuracy) print("Accuracy:",metrics.accuracy_score(final_prediction,y_test)) #evaluation(Confusion Metrix) print("Confusion Metrix:\n",metrics.confusion_matrix(final_prediction,y_test)) print() print("\t\taccuracy: {}".format(metrics.accuracy_score(final_prediction, y_test))) print(metrics.classification_report(final_prediction,y_test)) ==================================================== time cost: 1520.9758558273315 Accuracy: 0.8383838383838383 Confusion Metrix: [[40 3] [13 43]] accuracy: 0.8383838383838383 precision recall f1-score support 0 0.75 0.93 0.83 43 1 0.93 0.77 0.84 56 avg / total 0.86 0.84 0.84 99 SVM Stacking¶ In [31]: X_pred=np.array([svm_pred,knn_prediction,RF_prediction]).T #pred=np.array([pred1,pred2,pred3,pred4]).T #output layer containg y_test y_train #split dataset from sklearn.neural_network import MLPClassifier from sklearn.model_selection import train_test_split from sklearn.model_selection import GridSearchCV from time import * np.random.seed(123) # Set the parameters by cross-validation tuned_parameters = [{'kernel': ['rbf'], 'gamma': [1e-2, 1e-3, 1e-4, 1e-5], 'C': [0.001, 0.10, 0.1, 10, 25, 50, 100, 1000]}, {'kernel': ['sigmoid'], 'gamma': [1e-2, 1e-3, 1e-4, 1e-5], 'C': [0.001, 0.10, 0.1, 10, 25, 50, 100, 1000] },{'kernel': ['linear'], 'C': [0.001, 0.10, 0.1, 10, 25, 50, 100, 1000]}] print() grid_sv = GridSearchCV(SVC(), tuned_parameters, cv=10, scoring='accuracy') t0=time() grid_sv.fit(X_pred, y_train) t = time() - t0 print("Best Hyper Parameters:\n",grid_sv.best_params_) print(grid_sv.best_score_) Best Hyper Parameters: {'C': 10, 'gamma': 0.01, 'kernel': 'rbf'} 0.8558951965065502 In [32]: X_opt_pred=np.array([opt_svm_pred,opt_knn_pred,opt_RF_pred]).T #output layer containg y_test y_test final_prediction_sv=grid_sv.best_estimator_.predict(X_opt_pred) print("=" * 52) print("time cost: {}".format(t)) print() #importing the metrics module from sklearn import metrics #evaluation(Accuracy) print("Accuracy:",metrics.accuracy_score(final_prediction_sv,y_test)) #evaluation(Confusion Metrix) print("Confusion Metrix:\n",metrics.confusion_matrix(final_prediction_sv,y_test)) print() print("\t\taccuracy: {}".format(metrics.accuracy_score(final_prediction_sv, y_test))) print(metrics.classification_report(final_prediction_sv,y_test)) ==================================================== time cost: 2.2583255767822266 Accuracy: 0.8383838383838383 Confusion Metrix: [[40 3] [13 43]] accuracy: 0.8383838383838383 precision recall f1-score support 0 0.75 0.93 0.83 43 1 0.93 0.77 0.84 56 avg / total 0.86 0.84 0.84 99 RF Stacking¶ In [58]: X_pred=np.array([svm_pred,knn_prediction,RF_prediction]).T #pred=np.array([pred1,pred2,pred3,pred4]).T #output layer containg y_test y_train #split dataset from sklearn.neural_network import MLPClassifier from sklearn.model_selection import train_test_split from sklearn.model_selection import GridSearchCV from time import * np.random.seed(123) modelS=RandomForestClassifier() #hyper parameters set params = {'criterion':['gini','entropy'], 'n_estimators':[10,15,20,25,30], 'min_samples_leaf':[1,2,3], 'min_samples_split':[3,4,5,6,7], 'random_state':[123], 'n_jobs':[-1]} #Making models with hyper parameters sets RfgridS = GridSearchCV(modelS, param_grid=params,cv=10, n_jobs=-1) t0=time() RfgridS.fit(X_pred, y_train) t = time() - t0 print("Best Hyper Parameters:\n",RfgridS.best_params_) print(RfgridS.best_score_) Best Hyper Parameters: {'criterion': 'gini', 'min_samples_leaf': 2, 'min_samples_split': 3, 'n_estimators': 10, 'n_jobs': -1, 'random_state': 123} 0.851528384279476 In [39]: X_opt_pred=np.array([opt_svm_pred,opt_knn_pred,opt_RF_pred]).T #output layer containg y_test y_test final_prediction_rf=RfgridS.best_estimator_.predict(X_opt_pred) print("=" * 52) print("time cost: {}".format(t)) print() #importing the metrics module from sklearn import metrics #evaluation(Accuracy) print("Accuracy:",metrics.accuracy_score(final_prediction_rf,y_test)) #evaluation(Confusion Metrix) print("Confusion Metrix:\n",metrics.confusion_matrix(final_prediction_rf,y_test)) print() print("\t\taccuracy: {}".format(metrics.accuracy_score(final_prediction_rf, y_test))) print(metrics.classification_report(final_prediction_rf,y_test)) ==================================================== time cost: 149.77435994148254 Accuracy: 0.8383838383838383 Confusion Metrix: [[40 3] [13 43]] accuracy: 0.8383838383838383 precision recall f1-score support 0 0.75 0.93 0.83 43 1 0.93 0.77 0.84 56 avg / total 0.86 0.84 0.84 99 X_pred=np.array([svm_pred,knn_prediction,RF_prediction]).T #pred=np.array([pred1,pred2,pred3,pred4]).T #output layer containg y_test y_train #split dataset from sklearn.neural_network import MLPClassifier from sklearn.model_selection import train_test_split from sklearn.model_selection import GridSearchCV from time import * np.random.seed(123) modelS=RandomForestClassifier() #hyper parameters set params = {'criterion':['gini','entropy'], 'n_estimators':[10,15,20,25,30], 'min_samples_leaf':[1,2,3], 'min_samples_split':[3,4,5,6,7], 'random_state':[123], 'n_jobs':[-1]} #Making models with hyper parameters sets RfgridS = GridSearchCV(modelS, param_grid=params,cv=10, n_jobs=-1) t0=time() RfgridS.fit(X_pred, y_train) t = time() - t0 print("Best Hyper Parameters:\n",RfgridS.best_params_) print(RfgridS.best_score_) X_opt_pred=np.array([opt_svm_pred,opt_knn_pred,opt_RF_pred]).T #output layer containg y_test y_test final_prediction_rf=RfgridS.best_estimator_.predict(X_opt_pred) print("=" * 52) print("time cost: {}".format(t)) print() #importing the metrics module from sklearn import metrics #evaluation(Accuracy) print("Accuracy:",metrics.accuracy_score(final_prediction_rf,y_test)) #evaluation(Confusion Metrix) print("Confusion Metrix:\n",metrics.confusion_matrix(final_prediction_rf,y_test)) print() print("\t\taccuracy: {}".format(metrics.accuracy_score(final_prediction_rf, y_test))) print(metrics.classification_report(final_prediction_rf,y_test)) -
The input of the output layer in Matlab neural network toolbox
I have created the following network in Matlab. How can I get the output for a given input vector in a specific layer?
Specifically:
Output= net(Input);I want the input of the output layer without the activation function, I am interested in the results of X*W of the last layer before applying the sigmoid function.
Bests,
-
Gitlab CI Pipeline: Cannot create pods in the namespace
I have a kubernetes cluster (rancherOS & RKE) that has a running gitlab runner pod. Connection to my GitLab instance works fine.
If I activate the pipeline, it directly fails with this error:
Running with gitlab-runner 11.4.2 (cf91d5e1) on Kubernetes Runner e5e25776 Using Kubernetes namespace: gitlab-managed-apps Using Kubernetes executor with image ubuntu:latest ... ERROR: Job failed (system failure): pods is forbidden: User "system:serviceaccount:gitlab-managed-apps:default" cannot create pods in the namespace "gitlab-managed-apps"This here is my gitlab-runner deployment yaml:
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: gitlab-runner namespace: gitlab-managed-apps spec: replicas: 1 selector: matchLabels: name: gitlab-runner template: metadata: labels: name: gitlab-runner spec: containers: - args: - run image: gitlab/gitlab-runner:latest imagePullPolicy: Always name: gitlab-runner securityContext: privileged: true volumeMounts: - mountPath: /etc/gitlab-runner name: config - mountPath: /etc/ssl/certs name: cacerts readOnly: true restartPolicy: Always volumes: - configMap: name: gitlab-runner name: config - hostPath: path: /usr/share/ca-certificates/mozilla name: cacerts hostNetwork: trueI tried to add a security context with the parameter "privileged: true" but that does not help..
Has anyone an idea on how to grant the gitlab-runner deployment the right permission to create other pods in the namespace "gitlab-managed-apps"?
Thanks a lot :)
-
How can I convert ascii-art database table pictures into json?
This is a case of not-sure-what-to-search for. I have a program that displays mysql query outputs like this:
$ table +-----+-----+ | foo | bar | +-----+-----+ | 1 | a | +-----+-----+ | 2 | b | +-----+-----+I want to view this in JSON. Something like so:
$ table | tbr2j { "foo" : [1, 2] , "bar" : ["a", "b"] }Or like so:
$ table | tbc2j [ { "foo" : 1, "bar" : "a"} , { "foo" : 2, "bar" : "b"} ]Is there a tool that does this? If not I may write one, but I'd rather not reinvent the wheel if it exists. Also, if that table format has a name, I'd like to know it (then I'd know what to search for).
-
Bitbucket / I cannot see the artifacts in pipelines
I run e2e tests on CI environment, but I cannot see the artifacts in pipelines.
bitbucket-pipelines.yml: image: cypress/base:10 options: max-time: 20 pipelines: default: -step: script: - npm install -npm run test artifacts: -/opt/atlassian/pipelines/agent/build/cypress/screenshots/* -screenshots/*.png
Maybe I typed in the wrong way path, but I am not sure.
Does anyone have any ideas what I am doing wrong?