Mobile Edge Computing (MEC) has been envisioned as an efficient solution to provide computation-intensive yet latency-sensitive services for wireless devices. In this paper, we investigate the optimal dynamic spectrum allocation-assisted multiuser computation offloading in MEC for overall latency minimization. Specifically, we first focus on a static multiuser computation offloading scenario and jointly optimize users' offloading decisions, transmission durations, and Edge Servers' (ESs) resource allocations. Owing to the nonconvexity of our joint optimization problem, we identify its layered structure and decompose it into two problems: a subproblem and a top problem. For the subproblem, we propose a bisection search-based algorithm to efficiently find the optimal users' offloading decisions and ESs’ resource allocations under a given transmission duration. Second, we use a linear search-based algorithm for solving the top problem to obtain the optimal transmission duration based on the result of the subproblem. Further, after solving the static scenario, we consider a dynamic scenario of multiuser computation offloading with time-varying channels and workload. To efficiently address this dynamic scenario, we propose a deep reinforcement learning-based online algorithm to determine the near-optimal transmission duration in a real-time manner. Numerical results are provided to validate our proposed algorithms for minimizing the overall latency in both static and dynamic offloading scenarios. We also demonstrate the advantages of our proposed algorithms compared to the conventional multiuser computation offloading schemes.