Pass drag event from div to mapbox layer

Here is a jsfiddle I want to move the map, but as the div follows the mouse pointer looks like the event is catched by div. I want to pass this event to map when the mouse is over the div or buttons. I want only click event is available for buttons but drag is for map. I tried to use['pointer-events'] = 'none'; but it is kind of tricky for me. Is there any simple way? Thanks.

var voteDiv1 = document.getElementById('vote1');
map.on('mousemove', function (e) {
    var c = map.project(e.lngLat);
    console.log(c) = (c.x) -50 + 'px'; = (c.y) -50 + 'px';