Phantom hidden links

[MS]

Hi,

I've got an odd 'phantom' hidden links situation - I've a fix but I'd
like to understand what's happening.

Ok so I'm backing up a Win 7 PC using rsync over ssh by utilizing dll
and exe files from the Cygwin project.

When recursively backing up the folder "...\Users\Username\Documents\"
rsync also then backs up "My Music", "My Videos", and "My Pictures" but
placing each as a subdirectory of "Documents", EG.

"...\Users\Username\Documents\My Music\"
"...\Users\Username\Documents\My Pictures\"
"...\Users\Username\Documents\My Videos\"

But no link is shown to these 3 folders from within
"...\Users\Username\Documents\" !! I made sure that show hidden files
and folders was on but still nothing shown at all.

I've fixed this by using rsync's "--exclude" command but I'd like to
understand what's happening.

Where are these 'phantom hidden links' and how do I control (alter,
delete, etc.) them?

Thanks.

 

[VanguardLH]

Hi,

I've got an odd 'phantom' hidden links situation - I've a fix but I'd
like to understand what's happening.

Ok so I'm backing up a Win 7 PC using rsync over ssh by utilizing dll
and exe files from the Cygwin project.

When recursively backing up the folder "...\Users\Username\Documents\"
rsync also then backs up "My Music", "My Videos", and "My Pictures" but
placing each as a subdirectory of "Documents", EG.

"...\Users\Username\Documents\My Music\"
"...\Users\Username\Documents\My Pictures\"
"...\Users\Username\Documents\My Videos\"

But no link is shown to these 3 folders from within
"...\Users\Username\Documents\" !! I made sure that show hidden files
and folders was on but still nothing shown at all.

I've fixed this by using rsync's "--exclude" command but I'd like to
understand what's happening.

Where are these 'phantom hidden links' and how do I control (alter,
delete, etc.) them?

Thanks.

Are you perhaps following junction points when you are backing up? You
shouldn't since that results in duplicated files in the backup: once
when you visit the real folder and again when you follow the junction
point. Junction points are defined within NTFS, not as an object on the
hard disk.

http://en.wikipedia.org/wiki/Junction_point

Since you're using Cygwin, presumably you are familiar with *NIX and its
use of soft (or symbolic) and hard links. Junction points are like soft
links to directories. There are lots of utilities to create junction
points. I use [Junction] Link Magic (http://www.rekenwonder.com/).

You need to ask the author of your backup software if they know their
product explicitly ignores junction points. If not, you'll have to add
them as exclusions in your backup job. Since you are the author, you'll
have to do the exclusions on junction links.

 

[MS]

Hi,

I've got an odd 'phantom' hidden links situation - I've a fix but I'd
like to understand what's happening.

Ok so I'm backing up a Win 7 PC using rsync over ssh by utilizing dll
and exe files from the Cygwin project.

When recursively backing up the folder "...\Users\Username\Documents\"
rsync also then backs up "My Music", "My Videos", and "My Pictures" but
placing each as a subdirectory of "Documents", EG.

"...\Users\Username\Documents\My Music\"
"...\Users\Username\Documents\My Pictures\"
"...\Users\Username\Documents\My Videos\"

But no link is shown to these 3 folders from within
"...\Users\Username\Documents\" !! I made sure that show hidden files
and folders was on but still nothing shown at all.

I've fixed this by using rsync's "--exclude" command but I'd like to
understand what's happening.

Where are these 'phantom hidden links' and how do I control (alter,
delete, etc.) them?

Thanks.

Are you perhaps following junction points when you are backing up? You
shouldn't since that results in duplicated files in the backup: once
when you visit the real folder and again when you follow the junction
point. Junction points are defined within NTFS, not as an object on the
hard disk.

http://en.wikipedia.org/wiki/Junction_point

Since you're using Cygwin, presumably you are familiar with *NIX and its
use of soft (or symbolic) and hard links. Junction points are like soft
links to directories. There are lots of utilities to create junction
points. I use [Junction] Link Magic (http://www.rekenwonder.com/).

You need to ask the author of your backup software if they know their
product explicitly ignores junction points. If not, you'll have to add
them as exclusions in your backup job. Since you are the author, you'll
have to do the exclusions on junction links.

Thanks for the explanation VanguardLH.

Yes that's what's happening, I've never heard of junction points before
but as you supposed I am a UNIX man who's not over familiar with NTFS.

No need to contact the author (though I'll let the Cygwin project guys
know) cos as I said by using rsync's "--exclude" facility the backup is
working perfectly, I just wanted to understand what was going on, and
now -thanks to you- I do.

It's possible that one of rsync.exe's switches will be able to ignore
junction points, but rsync.exe's help is all in UNIX language and does
not mention junction points at all, so it's hard to tell which of the
symlink switches apply (if any). I'll investigate on the Cygwin mailing
list.

Thanks again.

 

[VanguardLH]

Hi,

I've got an odd 'phantom' hidden links situation - I've a fix but I'd
like to understand what's happening.

Ok so I'm backing up a Win 7 PC using rsync over ssh by utilizing dll
and exe files from the Cygwin project.

When recursively backing up the folder "...\Users\Username\Documents\"
rsync also then backs up "My Music", "My Videos", and "My Pictures" but
placing each as a subdirectory of "Documents", EG.

"...\Users\Username\Documents\My Music\"
"...\Users\Username\Documents\My Pictures\"
"...\Users\Username\Documents\My Videos\"

But no link is shown to these 3 folders from within
"...\Users\Username\Documents\" !! I made sure that show hidden files
and folders was on but still nothing shown at all.

I've fixed this by using rsync's "--exclude" command but I'd like to
understand what's happening.

Where are these 'phantom hidden links' and how do I control (alter,
delete, etc.) them?

Thanks.

Are you perhaps following junction points when you are backing up? You
shouldn't since that results in duplicated files in the backup: once
when you visit the real folder and again when you follow the junction
point. Junction points are defined within NTFS, not as an object on the
hard disk.

http://en.wikipedia.org/wiki/Junction_point

Since you're using Cygwin, presumably you are familiar with *NIX and its
use of soft (or symbolic) and hard links. Junction points are like soft
links to directories. There are lots of utilities to create junction
points. I use [Junction] Link Magic (http://www.rekenwonder.com/).

You need to ask the author of your backup software if they know their
product explicitly ignores junction points. If not, you'll have to add
them as exclusions in your backup job. Since you are the author, you'll
have to do the exclusions on junction links.

Thanks for the explanation VanguardLH.

Yes that's what's happening, I've never heard of junction points before
but as you supposed I am a UNIX man who's not over familiar with NTFS.

No need to contact the author (though I'll let the Cygwin project guys
know) cos as I said by using rsync's "--exclude" facility the backup is
working perfectly, I just wanted to understand what was going on, and
now -thanks to you- I do.

It's possible that one of rsync.exe's switches will be able to ignore
junction points, but rsync.exe's help is all in UNIX language and does
not mention junction points at all, so it's hard to tell which of the
symlink switches apply (if any). I'll investigate on the Cygwin mailing
list.

Thanks again.

But since Cygwin is from the *NIX world with which you are familiar,
you're saying that you couldn't find info on soft (symbolic) links in
rsync's manual pages? If not, I suspect it doesn't recognize a file
system based link.

Most utilities that I've seen (other than just command-line stuff for
*NIX) often warns you about possible side effects of using
soft/hard/junction links. Also, later when you restore you host from
the backups where these file system based links were defined, they don't
get restored which means you lose them in an image restore (and they
obviously won't be available in a logical file restore). You need to
ingore them in backups and then recreate them after a restore.

 

[Mark Blain]

[...]

Thanks for the explanation VanguardLH.

Yes that's what's happening, I've never heard of junction points before
but as you supposed I am a UNIX man who's not over familiar with NTFS.

No need to contact the author (though I'll let the Cygwin project guys
know) cos as I said by using rsync's "--exclude" facility the backup is
working perfectly, I just wanted to understand what was going on, and
now -thanks to you- I do.

It's possible that one of rsync.exe's switches will be able to ignore
junction points, but rsync.exe's help is all in UNIX language and does
not mention junction points at all, so it's hard to tell which of the
symlink switches apply (if any). I'll investigate on the Cygwin mailing
list.

Thanks again.

The rsync manpage does have a section about symbolic links. I tried a
quick web search for various combinations of keywords rsync, cygwin,
"symbolic link" and NTFS and found a few results relevant to your issue:

<http://lists.samba.org/archive/rsync/2008-August/021498.html>

<http://www.mail-archive.com/backuppc-
(e-mail address removed)/msg20728.html>

<http://www.itefix.no/i2/node/11771>

 

[MS]

MS wrote:

Hi,

I've got an odd 'phantom' hidden links situation - I've a fix but I'd
like to understand what's happening.

Ok so I'm backing up a Win 7 PC using rsync over ssh by utilizing dll
and exe files from the Cygwin project.

When recursively backing up the folder "...\Users\Username\Documents\"
rsync also then backs up "My Music", "My Videos", and "My Pictures" but
placing each as a subdirectory of "Documents", EG.

"...\Users\Username\Documents\My Music\"
"...\Users\Username\Documents\My Pictures\"
"...\Users\Username\Documents\My Videos\"

But no link is shown to these 3 folders from within
"...\Users\Username\Documents\" !! I made sure that show hidden files
and folders was on but still nothing shown at all.

I've fixed this by using rsync's "--exclude" command but I'd like to
understand what's happening.

Where are these 'phantom hidden links' and how do I control (alter,
delete, etc.) them?

Thanks.

Are you perhaps following junction points when you are backing up? You
shouldn't since that results in duplicated files in the backup: once
when you visit the real folder and again when you follow the junction
point. Junction points are defined within NTFS, not as an object on the
hard disk.

http://en.wikipedia.org/wiki/Junction_point

Since you're using Cygwin, presumably you are familiar with *NIX and its
use of soft (or symbolic) and hard links. Junction points are like soft
links to directories. There are lots of utilities to create junction
points. I use [Junction] Link Magic (http://www.rekenwonder.com/).

You need to ask the author of your backup software if they know their
product explicitly ignores junction points. If not, you'll have to add
them as exclusions in your backup job. Since you are the author, you'll
have to do the exclusions on junction links.

Thanks for the explanation VanguardLH.

Yes that's what's happening, I've never heard of junction points before
but as you supposed I am a UNIX man who's not over familiar with NTFS.

No need to contact the author (though I'll let the Cygwin project guys
know) cos as I said by using rsync's "--exclude" facility the backup is
working perfectly, I just wanted to understand what was going on, and
now -thanks to you- I do.

It's possible that one of rsync.exe's switches will be able to ignore
junction points, but rsync.exe's help is all in UNIX language and does
not mention junction points at all, so it's hard to tell which of the
symlink switches apply (if any). I'll investigate on the Cygwin mailing
list.

Thanks again.

But since Cygwin is from the *NIX world with which you are familiar,
you're saying that you couldn't find info on soft (symbolic) links in
rsync's manual pages? If not, I suspect it doesn't recognize a file
system based link.

No not at all. Rsync is very powerful it has a ton of symlink based
switches, soft/hard, follow/ignore, copy-as-link, etc. (google 'man
rsync'). The problem is that Junction Points are not the same as
symlinks -for starters they don't show up in a directory listing at all,
even as a hidden file/folder- and unsurprisingly due to it being a UNIX
program, it does not look like any of the rsync switches cater for JPs.
It's no problem because "--exclude" is working just fine, and I'm only
backing up user files, in the event of a full restoration being
necessary the OS will be restored separately which will handle
restoration of all JPs.

Cheers.

 

[MS]

[...]

Thanks for the explanation VanguardLH.

Yes that's what's happening, I've never heard of junction points before
but as you supposed I am a UNIX man who's not over familiar with NTFS.

No need to contact the author (though I'll let the Cygwin project guys
know) cos as I said by using rsync's "--exclude" facility the backup is
working perfectly, I just wanted to understand what was going on, and
now -thanks to you- I do.

It's possible that one of rsync.exe's switches will be able to ignore
junction points, but rsync.exe's help is all in UNIX language and does
not mention junction points at all, so it's hard to tell which of the
symlink switches apply (if any). I'll investigate on the Cygwin mailing
list.

Thanks again.

The rsync manpage does have a section about symbolic links. I tried a
quick web search for various combinations of keywords rsync, cygwin,
"symbolic link" and NTFS and found a few results relevant to your issue:

<http://lists.samba.org/archive/rsync/2008-August/021498.html>

<http://www.mail-archive.com/backuppc-
(e-mail address removed)/msg20728.html>

<http://www.itefix.no/i2/node/11771>

Thanks.

 

[VanguardLH]

The problem is that Junction Points are not the same as
symlinks -for starters they don't show up in a directory listing at all,
even as a hidden file/folder- and unsurprisingly due to it being a UNIX
program, it does not look like any of the rsync switches cater for JPs.

Something is goofy with your Windows setup. After creating 2 junction
points under My Documents folder, I can see them using:

Windows Explorer: They show as folders in the listing (I modify them to
use a different icon so they're obvious).

DOS shell: A 'dir' command shows them as folders and I can use a 'cd'
command to move into them (well, into the target folder at which the
junction points).

You sure this isn't a Cygwin defect in not listing junctions? It looks
like Cygwin took the lazy path to use .lnk files because those would
work under FAT and NTFS whereas reparse points are only definable within
NTFS. Yet both WE and the DOS shell can see .lnk files so it must be
some masking done by Cygwin.

 

[Alokasi]

Hidden Junction points

Although this is an old forum, I've just discovered it. The problem arises from some junctions that are also system files. They are created by the OS for backwards compatibility I believe. Go into a user's "Documents" directory with a command prompt and type "dir /AS" and you will see some of the directories that cause problems with rsync under Windows 7.