Algorithms for Factoring Polynomials over Finite Fields Tanja Lange 1 and Arne Winterhof 2 1 Institut f¨ ur Geometrie, TU Braunschweig, Pockelsstr. 14, D–38106 Braunschweig, Germany, ta.lange@tu-bs.de 2 Institut f¨ ur Diskrete Mathematik, ¨ Osterreichische Akademie der Wissenschaften, Sonnenfelsgasse 19, A–1010 Wien, Austria, arne.winterhof@oeaw.ac.at Abstract. Results on the worst case behavior of the authors’ extension (Theor. Comput. Sci. 234 (2000), 301-308) of Shoup’s algorithm for factoring polynomi- als over finite prime fields (Inf. Process. Lett. 33 (1990), 261-267) are improved. Moreover, the consequences of the average case behavior of the extended algorithm for multivariate algorithms are described, and an extension of Lenstra’s algorithm (Lond. Math. Soc, Lect. Note Ser. 154 (1990), 76-85) for root finding over finite prime fields is presented. Keywords. Factorization of polynomials, finite fields, deterministic algo- rithm 1 Introduction In [4] the authors analyze an extension of Shoup’s [7] deterministic algorithm for factoring polynomials over finite prime fields F p to arbitrary finite fields F q . In particular, for a polynomial of degree n over F q , q odd, they show that the fraction of all monic polynomials of degree n for which the extended algo- rithm fails to halt in O ( n 2+ε log 2 q ) F q -operations is O ( n 2 log 2 q/q ) . This is the first average case analysis of a deterministic algorithm over arbitrary F q of this quality. Unfortunately the worst case running time of the algorithm is O ( q 1/2 n 2+ε log q ) F q -operations. In Section 2 we recall the extended algorithm and its worst case analysis. In Section 3 we show that the worst case bound can be improved essentially un- der some certain restrictions. We show that the fraction of all monic polyno- mials of degree n for which the algorithm fails to halt in O ( n 2+ε r 2 p 1/2 log p ) is O ( n 2 /q ) if r is an odd integer satisfying r<p 1/2 /4, where q = p r is an odd prime power. Therefore, for a large fraction of polynomials the algorithm is slightly better than Shoup’s generalization of his algorithm to arbitrary finite fields ([8], [2]). (Shoup did not prove any bounds on the average case complex- ity. In the worst case his algorithm has a complexity of O ( (nr) 2+ε p 1/2 log p ) .) In Section 4 we describe the consequences of the average case bound of [4] for multivariate factorization.