Difference between tf.train.shuffle_batch_join and tf.train.shuffle_batch

See also questions close to this topic

• how to add non .py files into python egg

  I have a flask app which looks like

  my-app 
  │   └── src
  │       └── python
  │             └── config
  │             └── app
  │── MANIFEST.in
  └── setup.py
  

  The config folder is full of *.yaml files, I want to add all the static config files into my python egg after using

  python setup.py install
  

  My setup.py looks like

  import os
  from setuptools import setup, find_packages
  path = os.path.dirname(os.path.abspath(__file__))
  setup(
      name="app",
      version="1.0.0",
      author="Anna",
      description="",
      keywords=[],
      packages=find_packages(path + '/src/python'),
      package_dir={'': path + '/src/python'},
      include_package_data=True
  )
  

  I am trying the use the MANIFEST.in to add the config file However, it always give error

  error: Error: setup script specifies an absolute path:
  
  /Users/Anna/Desktop/my-app/src/python/app
  
  setup() arguments must *always* be /-separated paths relative to the
  setup.py directory, *never* absolute paths.
  

  I have not used any absolute paths in my code, I've seen other posts trying to bypass this error, by removing

  include_package_data=True
  

  However, in my case, if i do this to avoid this error, all my yamls won't be added.

  I was wondering if there are ways to fix this problem. Thanks
• How to extract all functions and API calls used in a Python source code?

  Let us consider the following Python source code;

  def package_data(pkg, roots):
      data = []
      for root in roots:
          for dirname, _, files in os.walk(os.path.join(pkg, root)):
              for fname in files:
                  data.append(os.path.relpath(os.path.join(dirname, fname), pkg))
  
      return {pkg: data}
  

  From this source code, I want to extract all the functions and API calls. I found a similar question and solution. I ran the solution given here and it generates the output [os.walk, data.append]. But I am looking for the following output [os.walk, os.path.join, data.append, os.path.relpath, os.path.join].

  What I understood after analyzing the following solution code, this can visit the every node before the first bracket and drop rest of the things.

  import ast
  
  class CallCollector(ast.NodeVisitor):
      def __init__(self):
          self.calls = []
          self.current = None
  
      def visit_Call(self, node):
          # new call, trace the function expression
          self.current = ''
          self.visit(node.func)
          self.calls.append(self.current)
          self.current = None
  
      def generic_visit(self, node):
          if self.current is not None:
              print("warning: {} node in function expression not supported".format(
                    node.__class__.__name__))
          super(CallCollector, self).generic_visit(node)
  
      # record the func expression 
      def visit_Name(self, node):
          if self.current is None:
              return
          self.current += node.id
  
      def visit_Attribute(self, node):
          if self.current is None:
              self.generic_visit(node)
          self.visit(node.value)  
          self.current += '.' + node.attr
  
  tree = ast.parse(yoursource)
  cc = CallCollector()
  cc.visit(tree)
  print(cc.calls)
  

  Can anyone please help me to modified this code so that this code can traverse the API calls inside the bracket?

  N.B: This can be done using regex in python. But it requires a lot of manual labors to find out the appropriate API calls. So, I am looking something with help of Abstract Syntax Tree.
• use correct version of 'pip' installed for your Python interpreter

  I am using pycharm , i am having this error by adding any package , Click Here

  i have tried lot of methods , but didn't succeed yet.

  Info :

  1. python 3.6.2
  2. pip 10.0.1
  3. VirtualEnv
• gcloud ml-engine with python3.5 tkinter import error on run

  I'm trying to run my model on Google cloud ml-engine with:

  gcloud ml-engine jobs submit training $NAME --module-name train.task_w2v \
       --package-path train --runtime-version 1.8 --python-version 3.5 \
       --scale-tier BASIC --staging-bucket $BUCKET --region $REGION
  

  And this is my setup.py:

  from setuptools import find_packages
  from setuptools import setup
  
  REQUIRED_PACKAGES = ['numpy', 'tensorflow', 'pandas', 'matplotlib',
  'opencv-python', 'PyYAML', 'coloredlogs', 'scikit-learn', 'scipy', 'matplotlib']
  
  setup(
      name='ConvMultiAttention',
      version='0.9',
      author='name',
      install_requires=REQUIRED_PACKAGES,
      packages=find_packages(),
      include_package_data=True,
  )
  

  The model runs fine locally and gets successfully build:

  I master-replica-0 Successfully installed model-0.9 coloredlogs-10.0 cycler-0.10.0 humanfriendly-4.15.1 kiwisolver-1.0.1 matplotlib-2.2.2 opencv-python-3.4.1.15 pyparsing-2.2.0 master-replica-0

  I master-replica-0 Running command: python3 -m train.task_w2v master-replica-0

  But then it gets this exception:

  master-replica-0 Traceback (most recent call last): File "/usr/lib/python3.5/tkinter/__init__.py", line 36, in import _tkinter ImportError: No module named '_tkinter'

  master-replica-0 Command '['python3', '-m', 'train.task_w2v']' returned non-zero exit status 1

  Since my understanding is that tkinter is part of python3.5 I don't really know what goes wrong here, or what to do. I tried to run it without matplotlib and with a lower tf version, but the problems consisted.

  Also I get these warnings:

  google-cloud-spanner 0.29.0 has requirement requests<3.0dev,>=2.18.4, but you'll have requests 2.13.0 which is incompatible.

  The script humanfriendly is installed in '/root/.local/bin' which is not on PATH.

  Which I don't really know how to handle, or if I even need to.
• Getting Youtube Favorites/Likes of a channel with API

  By using Youtube API, in Python 3.5.2, I want to fetch the likes/favorites of a giving channel id. Here the code I used:

  channel_details = youtube.channels().list(id=channel_id, part='snippet, contentDetails').execute()
      for c_detail in channel_details['items']:
          c_upload_list_id = c_detail['contentDetails']['relatedPlaylists']['Favorites']
  

  According the doc:https://developers.google.com/youtube/v3/docs/channels "favorites"/"Likes" properties are present in the "contentDetails" object. However, I'm getting the following error:

  KeyError: 'favorites'

  I get the same error when try to fetch the likes/favorites of my own Youtube Channel (even if they are set to public).

  Can someone help me ?

  Thanks
• Using Scipy's deconvolve function to deconvolve electrodermal activity data

  I wish to deconvolve an EDA (electrodermal activity) signal using a Bateman function as the filter as described here, using Scipy's deconvolve function.

  However, when I attempt this, the deconvolution graph does not look how I expect it to. Namely, it generally takes the shape of a mostly flat line, sometimes with spikes at multiples of the filter length:

  

  What am I missing here? Should I be smoothing the EDA curve? Am I hoping for too much from deconvolve? My code is below:

  import csv
  import numpy as np
  import matplotlib.pyplot as plt
  import scipy.signal as signal
  import math
  
  with open('test session 1.csv', newline='') as csvfile: 
      filereader = csv.reader(csvfile, delimiter=' ')
      i = 0
  
      timestamps = []
      conductances = []
  
      for row in filereader: 
          i += 1
  
          fields = ' '.join(row).split()
  
          if i > 3:
              timestamps.append(float(fields[0]))
              conductances.append(float(fields[5]))
  
      timestamps = [timestamp - timestamps[0] for timestamp in timestamps]
  
      c = 10.
      tau1 = 300
      tau2 = 2000
  
      bateman = [c * ( math.exp(-time / tau2) - math.exp(-time / tau1)) for time in timestamps]
      bateman = bateman[3:1700]
  
      deconv, remain = signal.deconvolve(conductances, bateman)
  
      fig, ax = plt.subplots(nrows=4)
  
      ax[0].plot(conductances, label="EDA Signal")
      ax[1].plot(bateman, label="Bateman Function")
      ax[2].plot(deconv, label="Deconvolution Result")
      ax[3].plot(remain, label="Remainder")
  
  
      for i in range(len(ax)):
          ax[i].legend(loc=4)
  
      plt.show()
  
• Jupyter does not show folder in my working directory

  I am running docker Jupiter notebook on a MacBook Pro. When starting Jupyter home only shows some of the folders in the working directory. When I cd to a folder and use it as the working directory I get the message "Notebook list is empty." See examples below.

  My directory:

  LewIss-MacBook-Pro:MyTensorFlow lewleib$ ls
  

  Gorner_tensorflow-mnist Tensor2018 models Gorner_tensorflow-rnn Untitled.ipynb tensorflow MyDeepTest generate_hmb3.py tensorflow-without-a-phd-master My_tensor1.html guided testgen NeuralNet1.ipynb install.sh README.md mnist

  One level down:

  LewIss-MacBook-Pro:MyDeepTest lewleib$ ls README.md guided models generate_hmb3.py install.sh testgen

  And one level more:

  LewIss-MacBook-Pro:guided lewleib$ ls
  

  chauffeur_guided.py epoch_guided.py ncoverage.py rambo_guided.py

  When I try and call Jupiter note book:

  LewIss-MacBook-Pro:guided lewleib$ docker run -it -p 8888:8888 -p 6006:6006 -v ~/lewleib/MyTensorFlow/MyDeepTest/guided:/notebooks tensorflow/tensorflow
  

  I get the following:

  guided Last Modified Name ..seconds ago The notebook list is empty.
• TensorFlow 1.3 ROCm port: cannot open '_pywrap_tensorflow_internal'

  In Ubuntu 16.04.4, I installed the TensorFlow 1.3 ROCm port (for an AMD Radeon RX Vega 64) according to the instructions starting at "Install required python packages" in

  https://github.com/ROCmSoftwarePlatform/tensorflow/blob/rocm-v1/rocm_docs/tensorflow-install-basic.md

  where I had previously installed ROCm from the AMD Debian repository according to the instructions in

  https://github.com/RadeonOpenCompute/ROCm

  Then, using pip to install the TF .whl package with no virtualization:

  $ wget http://repo.radeon.com/rocm/misc/tensorflow/tensorflow-1.3.0-cp27-cp27mu-manylinux1_x86_64.whl
  $ sudo python -m pip install tensorflow-1.3.0-cp27-cp27mu-manylinux1_x86_64.whl
  

  When I try to verify the installation using

  $ python -c "import tensorflow as tf; print(tf.__version__)"
  

  I get the following error:

  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/usr/local/lib/python2.7/dist-packages/tensorflow/__init__.py", line 24, in <module>
      from tensorflow.python import *
    File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/__init__.py", line 49, in <module>
      from tensorflow.python import pywrap_tensorflow
    File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 52, in <module>
      raise ImportError(msg)
  ImportError: Traceback (most recent call last):
    File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 41, in <module>
      from tensorflow.python.pywrap_tensorflow_internal import *
    File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
  _pywrap_tensorflow_internal = swig_import_helper()
    File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
  _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  ImportError: libCXLActivityLogger.so: cannot open shared object file: No such file or directory
  

  I verified that _pywrap_tensorflow_internal.so is present:

  $ find / -name \*pywrap\* -ls 2>/dev/null
   27526810      4 -rw-r--r--   1 root     staff        2558 Jul 20 11:41 /usr/local/lib/python2.7/dist-packages/tensorflow/python/pywrap_tensorflow.py
   27526811      4 -rw-r--r--   1 root     staff        1312 Jul 20 11:41 /usr/local/lib/python2.7/dist-packages/tensorflow/python/pywrap_tensorflow.pyc
   27526813     92 -rw-r--r--   1 root     staff       93912 Jul 20 11:41 /usr/local/lib/python2.7/dist-packages/tensorflow/python/pywrap_tensorflow_internal.pyc
   27526815 227172 -rwxr-xr-x   1 root     staff   232620600 Jul 20 11:41 /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so
   27526816     72 -rw-r--r--   1 root     staff       70386 Jul 20 11:41 /usr/local/lib/python2.7/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py
  

  Also checked my wheel and pip versions:

  $ pip list | grep wheel
  wheel                        0.29.0
  $ pip -V
  pip 10.0.1 from ---- python2.7/site-packages/pip (python 2.7)
  

  At first glance it looks as if some environmental variable is not set, so that _pywrap_tensorflow_internal.so is not being searched for on the correct path. Can anyone tell me if this is the case - or if the source of the problem is elsewhere? I did some searches and have come up essentially empty. Thanks in advance for any helpful responses.
• Calculate the gradients of the last state with respect to the initial state with GRU and understanding gradient tensor sizes in Tensorflow

  I have the following model in tensorflow:

  def output_layer(input_layer, num_labels):
      '''
      :param input_layer: 2D tensor
      :param num_labels: int. How many output labels in total? (10 for cifar10 and 100 for cifar100)
      :return: output layer Y = WX + B
      '''
      input_dim = input_layer.get_shape().as_list()[-1]
      fc_w = create_variables(name='fc_weights', shape=[input_dim, num_labels],
                              initializer=tf.uniform_unit_scaling_initializer(factor=1.0))
      fc_b = create_variables(name='fc_bias', shape=[num_labels], initializer=tf.zeros_initializer())
  
      fc_h = tf.matmul(input_layer, fc_w) + fc_b
      return fc_h
  
  def model(input_features):
  
      with tf.variable_scope("GRU_Layer1"):
          cell1 = tf.nn.rnn_cell.GRUCell(gru1_cell_size)
          # shape=(?, 64) ... gru1_cell_size=64
          initial_state1 = tf.placeholder(shape=[None, gru1_cell_size], dtype=tf.float32, name="initial_state1")
          output1, new_state1 = tf.nn.dynamic_rnn(cell1, input_features, dtype=tf.float32, initial_state=initial_state1)
  
      with tf.variable_scope("GRU_Layer2"):
          cell2 = tf.nn.rnn_cell.GRUCell(gru2_cell_size)
          # shape=(?, 32)...gru2_cell_size=32
          initial_state2 = tf.placeholder(shape=[None, gru2_cell_size], dtype=tf.float32, name="initial_state2")
          output2, new_state2 = tf.nn.dynamic_rnn(cell2, output1, dtype=tf.float32, initial_state=initial_state2)
  
      with tf.variable_scope("output2_reshaped"):
          # before, shape: (34, 100, 32), after, shape: (34 * 100, 32)
          output2 = tf.reshape(output2, shape=[-1, gru2_cell_size])
  
      with tf.variable_scope("output_layer"):
          # shape: (34 * 100, 3), num_labels=3
          predictions = output_layer(output2, num_labels)
          predictions = tf.reshape(predictions, shape=[-1, 100, 3])
      return predictions, initial_state1, initial_state2, new_state1, new_state2
  

  So as we can see from the code that the cell size of the first GRU is 64, the cell size of the second GRU is 32. And the batch size is 34 (but this is not important for me now). And the size of input features is 200. I have tried computing the gradients of the loss with respect to the trainable variables through:

  local_grads_and_vars = optimizer.compute_gradients(loss, tf.trainable_variables())
  # only the gradients are taken to add them later with the back propagated gradients from previous batch.
  local_grads = [grad for grad, var in local_grads_and_vars]
  
  for v in local_grads:
      print("v", v)
  

  After printing out the grads I got the following:

  v Tensor("Optimizer/gradients/GRU_Layer1/rnn/while/gru_cell/MatMul/Enter_grad/b_acc_3:0", shape=(264, 128), dtype=float32)
  v Tensor("Optimizer/gradients/GRU_Layer1/rnn/while/gru_cell/BiasAdd/Enter_grad/b_acc_3:0", shape=(128,), dtype=float32)
  v Tensor("Optimizer/gradients/GRU_Layer1/rnn/while/gru_cell/MatMul_1/Enter_grad/b_acc_3:0", shape=(264, 64), dtype=float32)
  v Tensor("Optimizer/gradients/GRU_Layer1/rnn/while/gru_cell/BiasAdd_1/Enter_grad/b_acc_3:0", shape=(64,), dtype=float32)
  v Tensor("Optimizer/gradients/GRU_Layer2/rnn/while/gru_cell/MatMul/Enter_grad/b_acc_3:0", shape=(96, 64), dtype=float32)
  v Tensor("Optimizer/gradients/GRU_Layer2/rnn/while/gru_cell/BiasAdd/Enter_grad/b_acc_3:0", shape=(64,), dtype=float32)
  v Tensor("Optimizer/gradients/GRU_Layer2/rnn/while/gru_cell/MatMul_1/Enter_grad/b_acc_3:0", shape=(96, 32), dtype=float32)
  v Tensor("Optimizer/gradients/GRU_Layer2/rnn/while/gru_cell/BiasAdd_1/Enter_grad/b_acc_3:0", shape=(32,), dtype=float32)
  v Tensor("Optimizer/gradients/output_layer/MatMul_grad/tuple/control_dependency_1:0", shape=(32, 3), dtype=float32)
  v Tensor("Optimizer/gradients/output_layer/add_grad/tuple/control_dependency_1:0", shape=(3,), dtype=float32)
  

  Here is the GRU cell from "Mastering TensorFlow 1.x: Advanced machine learning and deep learning concepts using TensorFlow 1.x and Keras" book. Here is the link as well: https://play.google.com/store/books/details?id=xtRJDwAAQBAJ&rdid=book-xtRJDwAAQBAJ&rdot=1&source=gbs_vpt_read&pcampaignid=books_booksearch_viewport

  So I was trying to understand the shapes of the gradients after printing out the tensors as shown in the code local_grads.

  

  From the GRU cell shown above I assumed that:

  1- Having a grad tensor with shape (264, 128) is used to calculate the activation before the input to r() and u(). If the output of r() and u() is 64, then there is a tensor of shape (128).

  2- Since the output size of the GRU is 64, therefore, I assumed that the input to the second layer GRU will of size 64 + 32(this is the cell size of the second GRU) which gives 96. Hence, similar to point 1, the gradient tensors will have shapes of (96, 64) and (64).

  3- Given that we have a dense layer after the second GRU layer, since the output is of size 3, then there is a gradient tensor for the corresponding weight of size (32, 3) and (3)

  My concern is why do we have tensors of shape (264, 64), (64) and (96, 32), (32).

  Second, Assume that I saved the gradients after training the model on the first batch, that is, after feeding a tensor of shape: (34, 100, 200) as input_features "In the model function argument", and output of shape (34 * 100, 3), how to back propagate these gradients on the second mini-batch?

  I would like to fix the gap as in the following image from https://deepmind.com/blog/decoupled-neural-networks-using-synthetic-gradients/:

  

  Where instead of having synthetic gradients, I would like to back propagate the gradients from the previous time step. So I was trying something like:

  prev_grads_val__ = tf.gradients([new_state1, new_state2], [initial_state1, initial_state2], grad_ys=previous_gradients)
  

  but this won't work giving the following error:

  ValueError: Passed 10 grad_ys for 2 ys
  

  And then prev_grads_val__ should be added to local_grads before performing the back propagation.

  Any help is much appreciated!!!
• How to train a machine learning algorithm to find this pattern: x1 < x2 without generating a new feature (e.g. x1-x2) first?

  If I had 2 features x1 and x2 where I know that the pattern is:

  if x1 < x2 then 
      class1 
  else 
      class2
  

  How can I train a machine learning algorithm to find such a pattern?

  I know that I could create a third feature x3 = x1-x2. Then feature x3 can easily be used by some machine learning algorithms. For example a decision tree can solve the problem 100% using x3 and just 3 nodes (1 decision and 2 leaf nodes).

  But, is it possible to solve this without creating new features?

  I tried MLP and SVM with different kernels, including svg kernel and the results are not great. This seems like a problem that should be easily solved 100% if a machine learning algorithm could only find such a pattern.

  As an example of what I tried, here is the scikit-learn code where the SVM could only get a score of 0.992:

  import numpy as np
  from sklearn.svm import SVC
  
  # Generate 1000 samples with 2 features with random values
  X_train = np.random.rand(1000,2)
  
  # Label each sample. If feature "x1" is less than feature "x2" then label as 1, otherwise label is 0.
  y_train = X_train[:,0] < X_train[:,1]
  y_train = y_train.astype(int) # convert boolean to 0 and 1
  
  svc = SVC(kernel = "rbf", C = 0.9) # tried all kernels and C values from 0.1 to 1.0
  
  svc.fit(X_train, y_train)
  print("SVC score: %f" % svc.score(X_train, y_train))
  

  Output running the code:

  SVC score: 0.992000
  

  This is an oversimplification of my problem. The real problem may have hundreds of features and different patterns, not just x1 < x2. However, to start with it would help a lot to know how to solve for this simple pattern.
• Best Machine Learning Algorithm to Rank Top 10 Items Based on Different Attributes?

  I am working on a project where I want to retrieve top n selections from a dataset based on different attributes. Let's say I am looking for the best store to buy a product. The algorithm will take in location, prices, closing/opening times, and return policy, and whether they have the product. It will return the top n (let's say 10) stores out of the dataset that it has found.

  I want to know what the best machine learning algorithm is for this scenario.
• How to consider word pairs/phrases for Word2Vec and other pre-processing

  So it's my first time using Word2Vec and Im using a wikipedia dump with WikiCorpus to pre-process the file before training my Word2Vec model. I want to use the following pre-processing techniques:

  1. Convert all letters to lowercase (I think WikiCorpus does this already).
  2. Remove all punctuation (Done by WikiCorpus).
  3. Consider word pairs/phrases as a single word, for example 'Big Apple' -> 'big_apple', not 'big', 'apple'.
  4. Convert all digits to their word forms, so '3' -> 'three'

  At the moment I have no idea how to do the last two. I know about num2text but not sure how to incorporate with WikiCorpus or Word2vec. Can anyone help?
• How to access underlying pipeline in Powershell Api C#

  I need to access underlying powershell default Pipeline which just created as result of Pwoershell.Create(); I need to stop that pipeline so that script execution must stopped immediately.

  I know I can Create Pipeline object and execute script by calling pipeline.Invoke(). But Pipeline object not giving me all Data streams which is being provided by Powershell instance.

  powerShellInstance = PowerShell.Create();
  powerShellInstance.Runspace = RunspaceFactory.CreateRunspace(iss)
  

  Later some point I need

  powerShellInstance.GetPipeline().Stop()  // something like this.
  

  Is there any thing by which I can achieve this?

  Edit

  For example I have following script.

  Set-LinkParameter"withError"  => This commandlet will throw error
  Set-LinkParameter11 "Mistyped" "error" => Powershell will throw an error of invalid commandlet
  
  Initialiize-Access  => this is invalid here and this commandlet will throw error.  Here it must not allow further script to execute.
  
  Write-Host "This is test message for host"
  Write-Information "test information"
  
  Write-Error "Access denied."  --> this is can be terminating error ( depends)
  Write-Warning "This is test warning." 
  Write-Verbose "This is verbose message"
  Write-Verbose "This is verbose message" -Verbose
  
  Write-Debug "Test Debug message" -Debug
  Write-Debug "wont appear at debug pipeline"
  
  Write-Output @(1,2,3) -NoEnumerate | measure
  
• Python: replacing shell pipeline with an ssh connection

  I have the following nice bash command :

  cat SomePythonScript.py | ssh remote_machine 'cat - | python'
  

  that works very fine and that I want to write in Python. I tried with 'subprocess' but did not get that far. Can someone help me ?

  from subprocess import  PIPE , Popen
  
  p1 = Popen(["cat ", "SomePythonScript.py"], stdout=PIPE)
  p2 = Popen(["remote_machine"], stdin=p1.stdout, stdout=PIPE)
  p3 = Popen(["cat -", "python"], stdin=p2.stdout, stdout=PIPE)
  
  p1.stdout.close()  
  p2.stdout.close()  
  
  output = p3.communicate()[0]
  

  I tried also with 2 processes/ pipes

  from  subprocess import PIPE , Popen  
  p1 = Popen([["cat", "SomePythonScript.py"], stdout=PIPE)
  p2 = Popen(["remote_machine","cat", "- python"], stdin=p1.stdout,    stdout=PIPE)
  p1.stdout.close() 
  output = p2.communicate()[0]
  

  I would be very glad to any help, suggestion , advices, explanation solution... Thk in advance
• General questions about the FDE Cycle

  1) when the instruction is predecoded and length decoded, the information is fed back to the fetch unit, the BTB uses this information to locate the 2-3 opcode bytes in parallel and works out the location of the address byte for each branching opcode found and then the address is analysed, I'd imagine that's how it works, correct me if I'm wrong, not sure how the predecode stage tags the bytes so this can happen and where the tags would be stored. 2) the BTB result is stored and tagged to the instruction in some way and whether it's speculated to be taken / not taken and the following instructions are then tagged in some way as speculative, how is this done / organised and where exactly is this tag stored 3) how exactly is this 'tag' checked on execution when the branch is taken / not taken and how is the branch result then returned to the BTB for updating, what orders / ensures / coordinates that process? 4) how are the instructions that depend on that branch prediction marked so that they can be flushed from the pipeline and if the branch is not taken? How is the address of the instruction that should have entered the pipeline then recovered so that the PC can be updated with it?

  5) instructions are fetched in 16 byte blocks, so 2 64 bit instruction addresses, assume that's one block a clock cycle, that means the PC needs to be incremented by 2 every cycle, what exactly increments the PC?, I assume some control unit or is the PC programmed to increment with every clock pulse, or some state machine when a new address arrives? And when the PC increments, how does the address immediately end up in the MAR and likewise what compares the MAR to the L1 instruction cache and says put result in MDR, if not go to MMU? I've been thinking perhaps it's a microoperation appended to every previous instruction decode or a separate sequence that repeats automatically in the control unit, but not sure. Wikipedia itself states 'logic in the CPU directs successive pulses to different places to perform a useful sequence'

  6) in the predecode stage, all 16 bytes are decoded in parallel rather than sequentially, which is faster, how does this work, how is an instruction length prefix identified to indeed be an instruction length prefix/mod/rm byte of an instruction and not mistaken for example for the SIB byte of an instruction, how does it find the start of the next instruction so quickly without using the length information of the previous instruction?. 'The level-1 code cache contains a considerable amount of predecode information. This includes information about where each instruction ends, where the opcode byte is, as well as distinctions between single, double and vector path instructions and identification of jumps and calls. Some of this information is copied to the level-2 cache, but not all. The low bandwidth for instructions coming from the level-2 cache may be due to the process of adding more predecode information', 7), how is the abovementioned predecode information accessed, how is it linked with the instruction it pertains to so that units in the CPU know how to find it with reference to an instruction?

  8) how does an instruction in the pipeline know that another instruction is actively going to commit to a memory location that is also referenced by the current instruction and that the result is dependent on it? How is the instruction then stalled for the exact right amount of cycles so that the result can be forwarded from the pipeline stage result register seamlessly?

  9) how are instructions at the destination of a predicted speculative jump prefetched into the L1 cache, or aren't they at all and it waits for a cache miss and then data comes in from a stream buffer?