React drag and drop not working as expected

I am making a restaurant reservation system and want my users to be able to add tables and place them on the canvas with drag and drop functionality. Now, When dragging the tables SLOWLY, it works, but as soon as it's done fast, they all snap to 0 on x axis and 0 on y axis, so in top left corner of the canvas. My code is as follows:

  const onMoveLabel = (event) => {
    event.target.removeEventListener("click", onLabelClick);
    event.target.dataset.hasMoved = true;

    let moveX;
    let moveY;

    moveX = event.clientX;
    moveY = event.clientY;
    let minBoundX = 0;
    let minBoundY = 0;
    let maxBoundX =
      minBoundX + canvasRef.current.offsetWidth - event.target.offsetWidth;
    let maxBoundY =
      minBoundY + canvasRef.current.offsetHeight - event.target.offsetHeight;

    const labels = document.querySelectorAll<HTMLDivElement>(".label");
    labels.forEach((label: HTMLDivElement) => {
      if (overlaps.includes(parseInt(label.dataset.id as string))) {
        label.style.left =
          Math.max(
            minBoundX,
            Math.min(label.offsetLeft + moveX - startX, maxBoundX)
          ) + "px";
        label.style.top =
          Math.max(
            minBoundY,
            Math.min(label.offsetTop + moveY - startY, maxBoundY)
          ) + "px";
      }
    });

    startX = moveX;
    startY = moveY;
  };
How many English words
do you know?
Test your English vocabulary size, and measure
how many words do you know
Online Test
Powered by Examplum