tf.train.replica_device_setter 'worker_device' parameter
According to the documentation https://www.tensorflow.org/api_docs/python/tf/train/replica_device_setter: By default, only Variable ops are placed on ps tasks, and the placement strategy is roundrobin over all ps tasks
.
Question: If 'worker_device' is supplied, then where do the 'Variable' ops live? On the worker or on ps.
See also questions close to this topic

Jupyter does not show folder in my working directory
I am running docker Jupiter notebook on a MacBook Pro. When starting Jupyter home only shows some of the folders in the working directory. When I cd to a folder and use it as the working directory I get the message "Notebook list is empty." See examples below.
My directory:
LewIssMacBookPro:MyTensorFlow lewleib$ ls
Gorner_tensorflowmnist Tensor2018 models Gorner_tensorflowrnn Untitled.ipynb tensorflow MyDeepTest generate_hmb3.py tensorflowwithoutaphdmaster My_tensor1.html guided testgen NeuralNet1.ipynb install.sh README.md mnist
One level down:
LewIssMacBookPro:MyDeepTest lewleib$ ls README.md guided models generate_hmb3.py install.sh testgen
And one level more:
LewIssMacBookPro:guided lewleib$ ls
chauffeur_guided.py epoch_guided.py ncoverage.py rambo_guided.py
When I try and call Jupiter note book:
LewIssMacBookPro:guided lewleib$ docker run it p 8888:8888 p 6006:6006 v ~/lewleib/MyTensorFlow/MyDeepTest/guided:/notebooks tensorflow/tensorflow
I get the following:
guided Last Modified Name ..seconds ago The notebook list is empty.

TensorFlow 1.3 ROCm port: cannot open '_pywrap_tensorflow_internal'
In Ubuntu 16.04.4, I installed the TensorFlow 1.3 ROCm port (for an AMD Radeon RX Vega 64) according to the instructions starting at "Install required python packages" in
where I had previously installed ROCm from the AMD Debian repository according to the instructions in
https://github.com/RadeonOpenCompute/ROCm
Then, using pip to install the TF .whl package with no virtualization:
$ wget http://repo.radeon.com/rocm/misc/tensorflow/tensorflow1.3.0cp27cp27mumanylinux1_x86_64.whl $ sudo python m pip install tensorflow1.3.0cp27cp27mumanylinux1_x86_64.whl
When I try to verify the installation using
$ python c "import tensorflow as tf; print(tf.__version__)"
I get the following error:
Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/local/lib/python2.7/distpackages/tensorflow/__init__.py", line 24, in <module> from tensorflow.python import * File "/usr/local/lib/python2.7/distpackages/tensorflow/python/__init__.py", line 49, in <module> from tensorflow.python import pywrap_tensorflow File "/usr/local/lib/python2.7/distpackages/tensorflow/python/pywrap_tensorflow.py", line 52, in <module> raise ImportError(msg) ImportError: Traceback (most recent call last): File "/usr/local/lib/python2.7/distpackages/tensorflow/python/pywrap_tensorflow.py", line 41, in <module> from tensorflow.python.pywrap_tensorflow_internal import * File "/usr/local/lib/python2.7/distpackages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module> _pywrap_tensorflow_internal = swig_import_helper() File "/usr/local/lib/python2.7/distpackages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description) ImportError: libCXLActivityLogger.so: cannot open shared object file: No such file or directory
I verified that _pywrap_tensorflow_internal.so is present:
$ find / name \*pywrap\* ls 2>/dev/null 27526810 4 rwrr 1 root staff 2558 Jul 20 11:41 /usr/local/lib/python2.7/distpackages/tensorflow/python/pywrap_tensorflow.py 27526811 4 rwrr 1 root staff 1312 Jul 20 11:41 /usr/local/lib/python2.7/distpackages/tensorflow/python/pywrap_tensorflow.pyc 27526813 92 rwrr 1 root staff 93912 Jul 20 11:41 /usr/local/lib/python2.7/distpackages/tensorflow/python/pywrap_tensorflow_internal.pyc 27526815 227172 rwxrxrx 1 root staff 232620600 Jul 20 11:41 /usr/local/lib/python2.7/distpackages/tensorflow/python/_pywrap_tensorflow_internal.so 27526816 72 rwrr 1 root staff 70386 Jul 20 11:41 /usr/local/lib/python2.7/distpackages/tensorflow/python/pywrap_tensorflow_internal.py
Also checked my wheel and pip versions:
$ pip list  grep wheel wheel 0.29.0 $ pip V pip 10.0.1 from  python2.7/sitepackages/pip (python 2.7)
At first glance it looks as if some environmental variable is not set, so that _pywrap_tensorflow_internal.so is not being searched for on the correct path. Can anyone tell me if this is the case  or if the source of the problem is elsewhere? I did some searches and have come up essentially empty. Thanks in advance for any helpful responses.

Calculate the gradients of the last state with respect to the initial state with GRU and understanding gradient tensor sizes in Tensorflow
I have the following model in tensorflow:
def output_layer(input_layer, num_labels): ''' :param input_layer: 2D tensor :param num_labels: int. How many output labels in total? (10 for cifar10 and 100 for cifar100) :return: output layer Y = WX + B ''' input_dim = input_layer.get_shape().as_list()[1] fc_w = create_variables(name='fc_weights', shape=[input_dim, num_labels], initializer=tf.uniform_unit_scaling_initializer(factor=1.0)) fc_b = create_variables(name='fc_bias', shape=[num_labels], initializer=tf.zeros_initializer()) fc_h = tf.matmul(input_layer, fc_w) + fc_b return fc_h def model(input_features): with tf.variable_scope("GRU_Layer1"): cell1 = tf.nn.rnn_cell.GRUCell(gru1_cell_size) # shape=(?, 64) ... gru1_cell_size=64 initial_state1 = tf.placeholder(shape=[None, gru1_cell_size], dtype=tf.float32, name="initial_state1") output1, new_state1 = tf.nn.dynamic_rnn(cell1, input_features, dtype=tf.float32, initial_state=initial_state1) with tf.variable_scope("GRU_Layer2"): cell2 = tf.nn.rnn_cell.GRUCell(gru2_cell_size) # shape=(?, 32)...gru2_cell_size=32 initial_state2 = tf.placeholder(shape=[None, gru2_cell_size], dtype=tf.float32, name="initial_state2") output2, new_state2 = tf.nn.dynamic_rnn(cell2, output1, dtype=tf.float32, initial_state=initial_state2) with tf.variable_scope("output2_reshaped"): # before, shape: (34, 100, 32), after, shape: (34 * 100, 32) output2 = tf.reshape(output2, shape=[1, gru2_cell_size]) with tf.variable_scope("output_layer"): # shape: (34 * 100, 3), num_labels=3 predictions = output_layer(output2, num_labels) predictions = tf.reshape(predictions, shape=[1, 100, 3]) return predictions, initial_state1, initial_state2, new_state1, new_state2
So as we can see from the code that the cell size of the first GRU is 64, the cell size of the second GRU is 32. And the batch size is 34 (but this is not important for me now). And the size of input features is 200. I have tried computing the gradients of the loss with respect to the trainable variables through:
local_grads_and_vars = optimizer.compute_gradients(loss, tf.trainable_variables()) # only the gradients are taken to add them later with the back propagated gradients from previous batch. local_grads = [grad for grad, var in local_grads_and_vars] for v in local_grads: print("v", v)
After printing out the grads I got the following:
v Tensor("Optimizer/gradients/GRU_Layer1/rnn/while/gru_cell/MatMul/Enter_grad/b_acc_3:0", shape=(264, 128), dtype=float32) v Tensor("Optimizer/gradients/GRU_Layer1/rnn/while/gru_cell/BiasAdd/Enter_grad/b_acc_3:0", shape=(128,), dtype=float32) v Tensor("Optimizer/gradients/GRU_Layer1/rnn/while/gru_cell/MatMul_1/Enter_grad/b_acc_3:0", shape=(264, 64), dtype=float32) v Tensor("Optimizer/gradients/GRU_Layer1/rnn/while/gru_cell/BiasAdd_1/Enter_grad/b_acc_3:0", shape=(64,), dtype=float32) v Tensor("Optimizer/gradients/GRU_Layer2/rnn/while/gru_cell/MatMul/Enter_grad/b_acc_3:0", shape=(96, 64), dtype=float32) v Tensor("Optimizer/gradients/GRU_Layer2/rnn/while/gru_cell/BiasAdd/Enter_grad/b_acc_3:0", shape=(64,), dtype=float32) v Tensor("Optimizer/gradients/GRU_Layer2/rnn/while/gru_cell/MatMul_1/Enter_grad/b_acc_3:0", shape=(96, 32), dtype=float32) v Tensor("Optimizer/gradients/GRU_Layer2/rnn/while/gru_cell/BiasAdd_1/Enter_grad/b_acc_3:0", shape=(32,), dtype=float32) v Tensor("Optimizer/gradients/output_layer/MatMul_grad/tuple/control_dependency_1:0", shape=(32, 3), dtype=float32) v Tensor("Optimizer/gradients/output_layer/add_grad/tuple/control_dependency_1:0", shape=(3,), dtype=float32)
Here is the GRU cell from "Mastering TensorFlow 1.x: Advanced machine learning and deep learning concepts using TensorFlow 1.x and Keras" book. Here is the link as well: https://play.google.com/store/books/details?id=xtRJDwAAQBAJ&rdid=bookxtRJDwAAQBAJ&rdot=1&source=gbs_vpt_read&pcampaignid=books_booksearch_viewport
So I was trying to understand the shapes of the gradients after printing out the tensors as shown in the code
local_grads
.From the GRU cell shown above I assumed that:
1 Having a grad tensor with shape
(264, 128)
is used to calculate the activation before the input tor()
andu()
. If the output ofr()
andu()
is 64, then there is a tensor of shape(128)
.2 Since the output size of the GRU is
64
, therefore, I assumed that the input to the second layer GRU will of size 64 + 32(this is the cell size of the second GRU) which gives96
. Hence, similar to point 1, the gradient tensors will have shapes of(96, 64)
and(64)
.3 Given that we have a dense layer after the second GRU layer, since the output is of size 3, then there is a gradient tensor for the corresponding weight of size
(32, 3)
and(3)
My concern is why do we have tensors of shape
(264, 64)
,(64)
and(96, 32)
,(32)
.Second, Assume that I saved the gradients after training the model on the first batch, that is, after feeding a tensor of shape:
(34, 100, 200)
asinput_features
"In the model function argument", and output of shape(34 * 100, 3)
, how to back propagate these gradients on the second minibatch?I would like to fix the gap as in the following image from https://deepmind.com/blog/decoupledneuralnetworksusingsyntheticgradients/:
Where instead of having synthetic gradients, I would like to back propagate the gradients from the previous time step. So I was trying something like:
prev_grads_val__ = tf.gradients([new_state1, new_state2], [initial_state1, initial_state2], grad_ys=previous_gradients)
but this won't work giving the following error:
ValueError: Passed 10 grad_ys for 2 ys
And then
prev_grads_val__
should be added tolocal_grads
before performing the back propagation.Any help is much appreciated!!!