This paper deals with the implementation of HMM based video classification algorithm using color feature vector on the open source BeagleBoard mobile platform. To simplify the development of video IO interfaces to the processor running the algorithm, we first choose the BeagleBoard-xM, a low-cost, lowpower, portable computer with a Cortex-A8 processor with a speed of 1GHz. The algorithm uses color feature vector with HMM as a classifier to classify videos into different genres. Video classification task can be often treated as a primary step for many other applications including data organization and maintenance, search, retrieval and so on. Most of the existing work includes only implementations on general purpose processors which are inadequate to meet the performance requirements of machine vision applications. For mobile platforms, the algorithms need to be implemented on embedded hardware to meet the requirements like size, power, cost etc. Various optimization techniques such as key frame extraction and feature extraction that are carried out to allow the execution of the algorithm are discussed. It further leads to efficient video browsing and retrieval strategies on mobile platforms. Experimental results obtained from the implementation of the video classification task on the ARM- based computing platform BeagleBoard-xM, showed that the classification efficiency of 89.33% was achieved.