Using Regex in Bash

[Xplorer4x4]

I always hated working with regex but seems to be the best tool for the job in this case, or atleast it seems like it to me. Please correct me if I am wrong. I want to be able to search a directory(but not subdirectories) for avi, mp4 or mkv files. If I stick to basic regex and search only for a single file type, say avi, it works. The problem is searching for multiple file types.

find . -prune -type f -regex '.*.avi'

Works fine.

These lines do not, but I am not sure why:

find . -prune -type f -regex (avi|mp4)
find . -prune -type f -regex (avi|mp4|mkv)
find . -prune -type f -regex (.*.avi|.*.mp4|.*.mkv)

[SecretCode]

Without fiddling with bash escapes, you could just use find's "or" capability, as in

find * -prune -type f -regex '.*log' -o -regex '.*py'

(find . -prune returns no results for me ... surely '.' is a directory and gets pruned before anything happens?)

I think what's happening with your syntax is that (avi|mp4) matches avip4 and avmp4, ie the | does not imply grouping of what's around it. No ... not that ... this seems to work:

find * -prune -type f -regex '.*log\|.*py'

Looks like you need to escape the | character even though it's quoted.

[Xplorer4x4]

Sorry I added prune manually to working code in the post. I had prune working, but forgot how now. My goal in using prune was to search . and . only excluding subdirectories. Maybe I misunderstood the basis of the prune command.

[SecretCode]

-prune might work if it was after the filters. Anyway, see my edit to my above post.

[Xplorer4x4]

Seems prune works fine in your edit:

find * -prune -type f -regex '.*avi\|.*mp4\|.*mkv'

I put an avi files in a subfolder of the working dir and named it exclude.avi so that if it showed up in the results, I would automaticly catch it.

I am curious, would "or" be lighter then using one long regex?

And finally, to piece this all together, I need to ftp it to my NAS, but I am a bit stumped on how to handle this via a service menu. I assume I am going to be best off including the find and ftp commands together in to a bash script.

nas.desktop file:

konsole --hold --workdir %f -e sh ~/.kde/share/kde4/services/ServiceMenus/wd-live/ftptowdlive-new.sh

nas-new.sh

find * -prune -type f -regex '.*avi\|.*mp4\|.*mkv' -exec ftp '192.168.1.160'

Now I realize the bash script is probably going to need a HereDoc, but for the time being, I am just trying to establish the basic ftp connection after the search but I get this

find: missing argument to `-exec'

I just want to see if I understand what this is saying, that it is expecting the results from find to be directed to the command after -exec?

So the final product needs to look something like this:

find * -prune -type f -regex '.*avi\|.*mp4\|.*mkv' -exec ftp 192.168.1.160 <<End-Of-Session
binary
cd /some/nas/folder/
put "{}"
bye
End-Of-Session

[SecretCode]

Seems prune works fine in your edit:

find * -prune -type f -regex '.*avi\|.*mp4\|.*mkv'

Yes, but I had to change the '.' to '*' for it to work.

On reflection I would have used -maxdepth 1 for this

find . -maxdepth 1 -type f -regex '.*avi\|.*mp4\|.*mkv'

I am curious, would "or" be lighter then using one long regex?

Lighter on my head to think about it, that's all!

And finally, to piece this all together, I need to ftp it to my NAS, but I am a bit stumped on how to handle this via a service menu. I assume I am going to be best off including the find and ftp commands together in to a bash script.

nas.desktop file:

konsole --hold --workdir %f -e sh ~/.kde/share/kde4/services/ServiceMenus/wd-live/ftptowdlive-new.sh

nas-new.sh

find * -prune -type f -regex '.*avi\|.*mp4\|.*mkv' -exec ftp '192.168.1.160'

Now I realize the bash script is probably going to need a HereDoc, but for the time being, I am just trying to establish the basic ftp connection after the search but I get this

find: missing argument to `-exec'

I just want to see if I understand what this is saying, that it is expecting the results from find to be directed to the command after -exec?

So the final product needs to look something like this:

find * -prune -type f -regex '.*avi\|.*mp4\|.*mkv' -exec ftp 192.168.1.160 <<End-Of-Session
binary
cd /some/nas/folder/
put "{}"
bye
End-Of-Session

When you use -exec you need to include the filename at some point using {} - and end it with \; (a ; escaped from the shell) - you don't have that in your first example. In the second, you don't have the \; - not even sure if a multi-line script can be passed to -exec.

Anyway, plain ftp is so old-fashioned. Why not use something like rsync? Saves all this 'binary' and 'cd' and 'bye'.

find . -maxdepth 1 -type f -regex '.*avi\|.*mp4\|.*mkv' -exec rsync {} 192.168.1.60:/some/nas/folder/ \;

(not tested)

or even

rsync *.{avi,mp4,mkv} 192.168.1.60:/some/nas/folder/

(also not tested)

[Xplorer4x4]

Ftp may be old fashioned but I have generally preferred it over sftp, scp, etc as the overhead has always hurt my thorough put substantially. Although it has been awhile since I tried sftp over the intranet or internet and not at all on my current isp so I will try to dig in to it tomorrow to see how big a difference it makes.

Sent from my DROID2 Global

[Xplorer4x4]

I tried both sets of code and they both fail. The first:

Exec=konsole --hold --workdir %f -e find . -maxdepth 1 -type f -regex '.*avi\|.*mp4\|.*mkv' -exec rsync {} 192.168.1.60:22/tmp/media/usb/USB2/c45f5474-ce19-4d5b-a483-672d1ace407f/1.New/ \;

Error:

Error processing Exec field in
The error comes in a pop up, not konsole.

The second:

Exec=konsole --hold --workdir %f -e rsync *.{avi,mp4,mkv} 192.168.1.160:/tmp/media/usb/USB2/c45f5474-ce19-4d5b-a483-672d1ace407f/1.New/

Error:

rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9]

In the case of the second example, I also noticed it prompts me for a password, but if I ssh in manually from konsole, it handles authentication via private/public keys just fine.

Besides authentication, I manually tested an SFTP transfer via Filezilla. Using SFTP I got speeds of less then 1MB/s. In fact it peaked at about 1MB/s. This is working with a roughly 1GB video file running about 45 min in length. At this speed, it will take roughly 30 minuets to transfer. When switching to FTP though, I see speeds of about 8MB/s average and peaks of 10MB/s(NAS is limited by a 100MB port). At this rate, SFTP is just unbearable.

[SecretCode]

Why are you running them via .desktop files? I can't see the advantage. If you need to, it may be simpler to put the entire command in a shell script and call that from the .desktop file.
eta - and definitely debug them from the console, not in desktop files

I'm really surprised at those ftp/sftp speed differences ... may have to do some tests.

[Xplorer4x4]

It keeps the dir much cleaner and organized to have it in a .desktop file then clog it up with a .desktop file and at least one bash script for each desktop file. In this case I would have a desktop file plus about 5 bash scripts for all 5 parent directories on the NAS.

As for ssh vs ftp speeds, let me know how your tests go. I realize sftp(and similar) will have some overhead because of the encryption, but this seems very extreme especially given we are talking about intranet speeds.