...
In pop up: open the file in pop up window (if possible with this file type). Many browsers block pop-up windows, so this is not a suggested approach.
Change file behaviors
To change the behavior of the file item, go into Edit Mode and Edit Settings for the file item. Under the Appearance section, change the Display option.
Suggested behavior
Generally, if the file is a PDF or other web-viewable file type, opening in a New Window is a good approach. Some files, such as Microsoft Word documents, cannot be opened in a web browser and will download no matter which option is selected. Some types of files, especially text-based files such as computer programs, may open in the web browser. For these files, Force Download, may be more appropriate.