If your iPod Touch / iPhone is not jailbroken, it is impossible to remove the stocks app.
If your iPod Touch / iPhone is jailbroken, you can use a program such as iFunBox or iPhoneBrowser to delete the Applications/Stocks.App folder. Restart your device and the Stocks app should be history.
Be warned though, if you do delete the Stocks app, you could potentially mess up your device to the point where you have to restore it.
Chat with our AI personalities
You cannot delete an app as it is downloading. You have to wait for it to finish downloading, and then you can delete it.
mmm not really
Hold down for about three seconds on an app and they will shake. Click the red X and it will delete. You can't do this on apps that came with the ipod. :)
If you remove a paid application you can simply re-download it free of charge from the iTunes App Store.
There is an App Store app on the iPod Touch that is used to download apps.