Why do we need userassist if we already have prefetch?

2018-10-09 16:25 Ariel Kalman imported from Stackoverflow

I know the purpose of both of the features but i don't exactly know the difference between them and what is the disadvantages and advantages of both of the features... Thanks!

See also questions close to this topic

powershell opens explorer window instead of launching the python interpreter
In powershell session typing python opens a Windows Explorer window "How do you want to open this file?" to select the program to open this file.

However the same python command works as expected (launches interpreter) in cmd.

The same issue when specifying python with a .py script name.

Do I need to fix something in registry keys or file associations? Why only powershell is behaving badly, but cmd works as expected?

EDIT:

Just resolved the issue:
```
PS > (get-command python.exe).Path
C:\Python\Python35_64b\python.exe
PS > (get-command python).Path
C:\Python\Python35_64b\Scripts\python
```
Apparently I had a file python without extension somehow stuck in one of the paths in %PATH% environment variable. After deleting this dummy file (how it got there?!) everything works fine.

Maybe this post still may help someone in the future (possibly myself)!

Still the question here is why cmd and powershell have a different order of searching for files in %PATH% with and without extensions (.exe)?

Is it possible to mix C++ OpenGL code with Java using LWJGL?

I want to make Java app. using CEF3 library. CEF is the library to embedded Google Chrome browser in any app. And LWJGL is used to write GL code in Java. But before using CEF, basic question is how to mix C++ and Java.

Java main calls C++ part as DLL
C++ part creates window and set up GL context
In the message loop, C++ call backs Java part again to do some GL work in Java.

Following test code fails with message:

FATAL ERROR in native method: Thread[main,5,main]: No context is current or a function that is not available in the current context was called. The JVM will abort execution.
at org.lwjgl.opengl.GL11.glColor3f(Native Method)
at Main.run(Main.java:18)
at Main.cppmain(Native Method)
at Main.main(Main.java:10)

Probably because Java part does not know about the GL context created by C++ part. My question is how can I set up GL context so that both C++ and Java may call GL functions?

Main.java

import org.lwjgl.opengl.GL11;

public class Main implements Runnable {
    {
        System.loadLibrary("cppgl");
    }

    public static void main(String[] args) {
        Main me = new Main();
        me.cppmain(me);
    }

    private native void cppmain(Runnable callback);

    @Override
    public void run() {
        // callback from cpp
        GL11.glColor3f(1.0f, 0.0f, 1.0f);
    }
}

cppgl.cpp

#pragma comment(lib, "OpenGL32.lib")

#include <windows.h>
#include <tchar.h>
#include <functional>
#include <gl/gl.h>
#include <jni.h>
#pragma comment(lib, "jvm.lib")

LRESULT CALLBACK WndProc(HWND hWnd, UINT mes, WPARAM wParam, LPARAM lParam) {
    if (mes == WM_DESTROY || mes == WM_CLOSE) { PostQuitMessage(0); return 0; }
    return DefWindowProc(hWnd, mes, wParam, lParam);
}

extern "C" JNIEXPORT void JNICALL Java_Main_cppmain
(JNIEnv *env, jobject, jobject callback) {

    LPCTSTR title = L"gl test";

    // prepare JNI call
    jclass cls = env->FindClass("Ljava/lang/Runnable;");
    jmethodID mid = env->GetMethodID(cls, "run", "()V");

    // window class
    HINSTANCE hInstance = ::GetModuleHandle(NULL);
    WNDCLASSEX wcex;
    ZeroMemory(&wcex, sizeof(wcex));
    wcex.cbSize = sizeof(wcex);
    wcex.style = CS_OWNDC | CS_HREDRAW | CS_VREDRAW;
    wcex.lpfnWndProc = WndProc;
    wcex.hInstance = hInstance;
    wcex.hbrBackground = (HBRUSH)(COLOR_WINDOW + 1);
    wcex.lpszClassName = title;
    if (!RegisterClassEx(&wcex)) return;

    // window
    RECT R = { 0, 0, 640, 480 };
    AdjustWindowRect(&R, WS_OVERLAPPEDWINDOW, FALSE);
    HWND hWnd = CreateWindow(title, title, WS_OVERLAPPEDWINDOW,
        CW_USEDEFAULT, 0, R.right - R.left, R.bottom - R.top,
        NULL, NULL, hInstance, NULL);
    if (!hWnd) return;

    // GL context
    PIXELFORMATDESCRIPTOR pfd;
    ZeroMemory(&pfd, sizeof(pfd));
    pfd.nSize = sizeof(pfd);
    pfd.nVersion = 1;
    pfd.dwFlags = PFD_DRAW_TO_WINDOW | PFD_SUPPORT_OPENGL | PFD_DOUBLEBUFFER;
    pfd.iPixelType = PFD_TYPE_RGBA;
    pfd.cColorBits = 32;
    pfd.cDepthBits = 24;
    pfd.iLayerType = PFD_MAIN_PLANE;
    HDC dc = GetDC(hWnd);
    int format = ChoosePixelFormat(dc, &pfd);
    SetPixelFormat(dc, format, &pfd);
    HGLRC glRC = wglCreateContext(dc);
    if (!wglMakeCurrent(dc, glRC)) return;

    // message loop
    ShowWindow(hWnd, SW_SHOW);
    MSG msg;
    do {
        if (PeekMessage(&msg, NULL, 0, 0, PM_REMOVE)) {
            TranslateMessage(&msg);
            DispatchMessage(&msg);
        }
        else {
            wglMakeCurrent(dc, glRC);
            glClearColor(0.0f, 0.5f, 1.0f, 1.0f);

            // call Java and change clear color here
            env->CallVoidMethod(callback, mid);

            glClear(GL_COLOR_BUFFER_BIT);
            glRectf(-0.5f, -0.5f, 0.5f, 0.5f);
            glFlush();
            SwapBuffers(dc);
            wglMakeCurrent(NULL, NULL);
        }
    } while (msg.message != WM_QUIT);

    // clean up
    wglMakeCurrent(NULL, NULL);
    wglDeleteContext(glRC);
    ReleaseDC(hWnd, dc);
}

How to find the original path of a symlink in Windows?

There are ways for Linux, but I did not find one for Windows. Thanks!
Serializing a complex, nested, user-defined object into a string in javascript
I am working with the following complicated object:
```
ObjectA {
    somearr: [
        ObjectB {
           somevar: ObjectD {...},
           somecircularvar: pointer
        },
        ObjectC {...},
        ...
    ],
    someobj: ObjectE {
        someothercircularvar: pointer,
        somevar: ObjectF {...}
    },
    someMethod: ...
}
```
The above object has the following:
- Nested objects
- Circular references to many locations within the object (not just the main reference)
- User-defined objects
Main Question: How do I turn this object into a string for storage, and how do I parse the string back into the object with all of the methods and variables as is?

Things I have Tried:
- Libraries
  - Flatted.js
  - Cyro.js
  - JSONfn.js
- Searches
  - Existing Stack Overflow Questions (none seemed to deal with my monstrosity)
  - Google searching serialization of user-defined objects (these could not deal with circular)
After trying these I saw that all of the "solutions" deal with circular and nested objects, not user-defined objects.

I do remember trying a couple other libraries but none could deal with circular, nested, and user-defined objects all at the same time.

The closest I have gotten to is the following parse:
```
{
    somearr: [
        {
           somevar: {...},
           somecircularvar: pointer
        },
        {...},
        ...
    ],
    someobj: {
        someothercircularvar: pointer, // Circular pointer conserved
        somevar: {...}
    }
}
```
Notice how my object names have disappeared, and each __proto__ is now the default object (not my object as defined by my local classes) with none of my methods conserved.

Huge thanks in advance to the person that can solve this problem.

Object.values not supported in IE - [solution] Object.keys

So since Object.values is not supported in IE, this is what I did to get it working:

const activityType = Object.keys(activityType).map((k) => activityType[k]);
const qualityType = Object.keys(qualityType).map((k) => qualityType[k]);

objTypes.forEach(activityType => {
   viewTypes.forEach(qualityType => {
      qna[activityType + qualityType] =  activityType + '.' + qualityType;
   });
});

Works fine on IE now. My question is for suggestions on cleaner/sleeker implementation.

Receiving Promise <Pending> instead of value
I have a a object that when I'm printing that it's returning Promise <Pending> (I've checked the type of getRateable and it is object)
```
getRateable = getRateableEntitiesTx(tx, hashtagList);
```
I can't have access to the value by this :
```
getRateableEntitiesTx(tx, hashtagList).then((res) => {return res})
```
If it's a Promise why it's not returning the res properly?

Thanks in advance for help

Prefetch Text Layout in RecyclerView, cannot resolve method setTextFuture

I'm trying to add Prefetch Text Layout to my reyclerview as shown here and here. Weirdly, it is only working on a my custom TextView and not with standard TextViews. For my standard TextView, I get Cannot resolve method setTextFuture error.

Also to add, everything was working fine before. the standard setText method worked for either custom or regular TextView. I have a total of four (4) TextViews, and all return the same error. I cast one as a custom TextView (even though it wasn't) and the error went away.

Here's my cleaned up code.

@Override
public void onBindViewHolder(@NonNull RecyclerViewHolder rvh, int position){
    rvh.setMtbCard(listFiltered_main.get(position));
}

public class RecyclerViewHolder extends RecyclerView.ViewHolder {
    public TextView tv_name;
    public LengthTextView tv_length;

    public RecyclerViewHolder(final View itemView, final int viewType){
        super(itemView);

        tv_name =itemView.findViewById(R.id.tv_name);
        tv_length =itemView.findViewById(R.id.tv_length);

    public void setMtbCard(final TrailsObject trailsObject) {

        //not working... getting "Cannot resolve method setTextFuture" error
        tv_name.setTextFuture(PrecomputedTextCompat.getTextFuture(
                trailsObject.getName_diff(),
                tv_name.getTextMetricsParamsCompat(),
                null)
        );

        //seemingly works fine
        tv_length.setTextFuture(PrecomputedTextCompat.getTextFuture(
                trailsObject.getLength_mi(),
                tv_length.getTextMetricsParamsCompat(),
                null)
        );
    }
}

UPDATE I noticed my custom TextView extends AppCompatTextView and not TextView, so I cast my other textviews as such and the errors went away and the code is running.

In my adapter

import android.support.v7.widget.AppCompatTextView;
public AppCompatTextView tv_name;

In my XML

<android.support.v7.widget.AppCompatTextView
        android:id="@+id/tv_mtb_name"
        style="@style/mtb_name"
        />

mxnet: how to set up a custom mx.io.DataIter to prefetch?

My mxnet script is likely limited by i/o of data loading into the GPU, and I am trying to speed this up by prefetching. The trouble is I can't figure out how to prefetch with a custom data iterator.

My first hypothesis/hope was that it would be enough to set the values of self.preprocess_threads and self.prefetch_buffer, as I had seen here for iterators such as mxnet.io.ImageRecordUInt8Iter. However, when I did this I saw no performance change relative to the script before I had set these variables, so clearly setting these did not work.

Then I noticed, the existence of a class mx.io.PrefetchingIter in addition to the base class for which I had implemented a child class mx.io.DataIter. I found this documentation, but I have not been able to find any examples, and I am a little confused about what needs to happen where/when. However, I am not clear on how to use this. For example. I see that in addition to next() it has an iter_next() method, which simply says "move to the next batch". What does this mean exactly? What does it mean to "move" to the next batch without producing it? I found the source code for this class, and based on a brief reading, it seems as though it takes multiple iterators and creates one thread per iterator. This likely would not work for my current design, as I really want multiple threads used to prefetch from the same iterator.

Here is what I am trying to do via a custom data iterator

I maintain a global multiprocessing.Queue on which I pop data as it becomes available
I produce that data by running (via multiprocessing) a command line script that executes a c++ binary which produces a numpy file
I open the numpy file and load its contents into memory, process them, and put the processed bits on the global multiprocessing.Queue
My custom iterator pulls off this queue and also kicks off more jobs to produce more data when the queue is empty.

Here is my code:

def launchJobForDate(date_str):
### this is a function that gets called via multiprocessing
### to produce new data by calling a c++ binary
### whenever data queue is empty so that we need to produce more data
    try:
        f = "testdata/data%s.npy"%date_str
        if not os.path.isfile(f):
            cmd = CMD % ( date_str, JSON_FILE, date_str, date_str, date_str)
            while True:
                try:
                    output = subprocess.check_output(cmd, shell=True)
                    break
                except:
                    pass
        while True:
            try:
                d = np.load(f)
                break
            except:
                pass
        data_queue.put((d, date_str))
    except Exception as ex:
        print("launchJobForDate: ERROR ", ex)

class ProduceDataIter(mx.io.DataIter):
    @staticmethod
    def processData(d, time_steps, num_inputs):
       try: 
            ...processes data...
            return [z for z in zip(bigX, bigY, bigEvalY, dates)]
        except Exception as ex:
            print("processData: ERROR ", ex)

    def __init__(self, num_mgrs, end_date_str):
        ## iter stuff
        self.preprocess_threads = 4
        self.prefetch_buffer = 1

        ## set up internal data to preserve state
        ## and make a list of dates for which to run binary

    @property
    def provide_data(self):
        return [mx.io.DataDesc(name='seq_var', 
                               shape=(args_batch_size * GPU_COUNT, 
                                      self.time_steps, 
                                      self.num_inputs), 
                               layout='NTC')]

    @property
    def provide_label(self):
        return [mx.io.DataDesc(name='bd_return', 
                                shape=(args_batch_size * GPU_COUNT)),             
                mx.io.DataDesc(name='bd_return', 
                                shape=(args_batch_size * GPU_COUNT, num_y_cols)), 
                mx.io.DataDesc(name='date', 
                               shape=(args_batch_size * GPU_COUNT))]                 


    def __next__(self):
        try:
            z = self.z.pop(0)       
            data = z[0:1]
            label = z[1:]
            return mx.io.DataBatch(data, label) 
        except Exception as ex:
            ### if self.z (a list) has no elements to pop we need
            ### to get more data off the queue, process it, and put it
            ### on self.x so it's ready for calls to __next__()
            while True:
                try:
                    d = data_queue.get_nowait()
                    processedData = ProduceDataIter.processData(d, 
                                                            self.time_steps, 
                                                            self.num_inputs)
                    self.z.extend(processedData)
                    counter_queue.put(counter_queue.get() - 1)

                    z = self.z.pop(0)
                    data = z[0:1]
                    label = z[1:]
                    return mx.io.DataBatch(data, label)

                except queue.Empty:
                    ...this is where new jobs to produce new data and put them 
                    ...on the queue would happen if nothing is left on the queue

I have then tried making one of these iterators as well as a prefetch iterator like so:

mgr      = ProcessMgr(2, end_date_str)
mgrOuter = mx.io.PrefetchingIter([mgr])

The problem is that mgrOuter immediately throws a StopIteration as soon as __next__() is called the first time, and without invoking mgr.__next__() as I thought it might.

Finally, I also noticed that gluon has a DataLoader object which seems like it might handle prefetching, however in this case it also seems to assume that the underlying data is from a Dataset which has a finite and unchanging layout (based on the fact that it is implemented in terms of getitem, which takes an index). So I have not pursued this option as it seem unpromising given the dynamic queue-like nature of the data I am generating as training input.

My questions are:

How do I need to modify my code above so that there will be prefetching for my custom iterator?
Where might I find an example or more detailed documentation of how mx.io.PrefetchingIter works?
Are there other strategies I should be aware of for getting more performance out of my GPUs via a custom iterator? Right now they are only operating at around 50% capacity, and upping (or lowering) the batch size doesn't change this. What other knobs might I be able to turn to increase GPU use efficiency?

Thanks for any feedback and advice.

Pre-fetch api data in Vuex with Nuxt.js

I am trying to pre-fetch some data and update Vuex before client-side kicks in.

store/index.js

export const state = () => ({});
export const getters = {};
export const actions = {
  async nuxtServerInit ({ dispatch }) {
    await dispatch('nasa/getImages');
  }
};

store/moduleName.js

import fetch from 'node-fetch';

export const state = () => ({
  images: []
});

export const mutations = {
  storeImages(state, data) {
    state.images = [];
    state.images.push(...data);
    console.log(state.images[0]); <- this logs in the terminal
  }
}

export const actions = {
  getImages(store) {
    return fetch('api/url').then(response => {
      response.json().then(function(data) {
          store.commit('storeImages', data.collection.items.slice(0, 24));
      });
    });
  }
}

My mutation gets triggered by nuxtServerInit and I am getting the first element logged in the terminal on page load. My store in the client-side however, is empty.

What am I missing?

Is there any Android deleted file log?

I have to report all deleted files included in a range of time.. So my question: Is there any record of all user-deleted files? I searched on web but no results.
Algorithm Fingerprints to find development source

I am curious if can algorithms be examined to identify potential signatures that might be associated with specific developers/countries? If so, how can it be possible to do so in theory? Also any sources to support the response would be cool.

Thanks!
Digital Forensic - media sterilise

I am done with my media and i have sterilise my media using sterilise. I just want to check if anyone know of any test/ways/procedure to confirmed that the media is sterilised.

Thanks

Regards J

Related questions

Why do we need userassist if we already have prefetch?

See also questions close to this topic