It is quite common to receive a text file that contains duplicate lines, which need to be removed. While the duplicate lines could be removed manually, this is a slow and error-prone process. Neovim's command-line mode allows us to leverage a wide range of command-line tools while editing files, and in this example we will use the uniq command to remove the duplicate lines.
Suppose we have the following file in a buffer and want to remove duplicates:
As a quick review of the steps required to invoke external commands from within Neovim, we first
enter command-line mode by pressing the : key, then define our command using the following pattern:
:[range]!{cmd} {args}
We can specify any valid range, such as a visual selection, but in this case we want to apply
the command to the entire file, so we specify the range as %.
As we reviewed in the command-line section, we external commands are called by
prefixing the command name with !, as in:
!{cmd}
In this case we want to use the uniq command to do the heavy lifting, so our call signature would be:
!uniq
uniq follows the call signature:
uniq {options} {input {output}}
When no input sources are specified, uniq takes its input from stdin and, likewise,
when no output is specified uniq directs its output to stdout, which is exactly what
we want to happen when we are executing external commands from command-line mode.
So, putting that all together, our first command will be:
:%!uniq
Executing this command produces the following output:
This helped, but isn't exactly what we were looking for. It turns out that by default the uniq command only removes adjacent repeating lines, so while we removed one instance of apple it still appears twice in the output. In order to
There are a few ways we could approach this. First, can create a simple pipeline by first sorting the lines then applying the uniq command, like this:
:%!sort|uniq
which achieves out goal of removing all of the duplicate lines of text:
We should note that uniq offers a variety of options such as the -d/--repeated option,
which performs the opposite function - returning only the duplicate lines, rather then filtering
them out. Starting from the original buffer we can execute this command to see the result:
Learn more about the uniq command on our uniq page, or check out the help pages for more information.