The easieast way is to check the base of the tail in front of the tail fin. If there are sharp bony plates on the sides of the tail you've got yourself a jack. Smooth sides on the tail like a trout or red and it's a pompano. As for the color thing, it depends on the water they come out of and it is not dependable. One of my articles in Gulf Coast Fisherman (The Bay Naturalist, spring 2006 I think) a couple of years ago included photos of a pompano, permit and a jack crevalle all caught on the same trip down in South Bay at SPI. That pompano was golden yellow especially around the throat, but yeah in murkier water they will be mostly silver.
And, small jacks are not bad at all if you grill them. If you like spanish mackerel you would be pretty happy with a 2 pound or so jack over the coals. The bigger they get, they fishier and even I won't mess with one more than about 5 pounds. They're no pompano, that's for sure but they don't deserve the food rep of a hardhead either.