Pass drag event from div to mapbox layer

Here is a jsfiddle https://jsfiddle.net/jes4fggd/. I want to move the map, but as the div follows the mouse pointer looks like the event is catched by div. I want to pass this event to map when the mouse is over the div or buttons. I want only click event is available for buttons but drag is for map. I tried to use voteDiv1.style['pointer-events'] = 'none'; but it is kind of tricky for me. Is there any simple way? Thanks.

var voteDiv1 = document.getElementById('vote1');
map.on('mousemove', function (e) {
    var c = map.project(e.lngLat);
    console.log(c)
    voteDiv1.style.left = (c.x) -50 + 'px';
    voteDiv1.style.top = (c.y) -50 + 'px';
});