Custom Color of Priority Tasks Change from Defaults

How do I get todo.txt vim plugin to respect the colours I have defined within my todo.cfg file?

I have custom color settings in todo.cfg for when doing various sorting/filtering on the command line but all my priority colours are changed when I open my todo.txt file with vim and this wonderful plugin. 