deleted files

As I understand NTFS, once a file is deleted then removed from the recycle
bin it is not recoverable by ordinary means.
Therer may be utilities out there to do that, and certainly after a defrag
it will likely be gone. No problem there, but are files recoverable?

Problem is the other way around, here. I deleted a set of files, emptied
recycle bin, deleted more the next day and emptied again. Today the first
set of files were back in the folder that had been deleted from. Only
explanation I can think of is they there was a brief power outage that day,
so the system went down and restarted. I remember win98 and 95 used to do
these automatic registry restores periodically, but I didn't think that
deleted files and former folder contents were stored in the registry.

From a security point of view - does this mean that deleted files, with
recycle bin emptied, are not really deleted?

Thanks

Stuart

Stuart Miller came up with this when s/he headbutted the keyboard a moment
ago in alt.computer.security:

> As I understand NTFS, once a file is deleted then removed from the recycle
> bin it is not recoverable by ordinary means.

Sort of. Read below.

> Therer may be utilities out there to do that, and certainly after a defrag
> it will likely be gone. No problem there, but are files recoverable?
>
> Problem is the other way around, here. I deleted a set of files, emptied
> recycle bin, deleted more the next day and emptied again. Today the first
> set of files were back in the folder that had been deleted from. Only
> explanation I can think of is they there was a brief power outage that
day,
> so the system went down and restarted. I remember win98 and 95 used to do
> these automatic registry restores periodically, but I didn't think that
> deleted files and former folder contents were stored in the registry.
>

they're not - FAT file systems have two FAT tables. One's a backup; not that
that does much good, because it's essentially a mirror of the primary.
NTFS uses journaling and backgrounding to give the illusion of a faster
filesystem. Now, the backgrounding (which makes heavy use of the large
caches found on very modern drives) isn't much use to you if there's a
power cut or if you're running a PVR on your system (you need realtime
writing to disk - no caching), however the journal is where you become
unstuck from a security viewpoint.

> From a security point of view - does this mean that deleted files, with
> recycle bin emptied, are not really deleted?
>

answer: deleting a file on an NTFS filesystem merely removes it from the
current journal. The file is still physically on the drive. The allocated
space is flagged for overwriting and bumped to the back of the write queue,
where it is forgotten about, until it reaches the front of the write queue
and is overwritten. On an average system, this can take /months/
considering light usage (browsing, writing documents, etc). On a heavy-use
system (such as a PVR) this can take a few days. Or even a few hours. Even
then the chances of that space being entirely overwritten in order are
fairly remote, so something of the original file will remain - very likely
enough to use as evidence after a forensic search.

To expand: a normal format does not erase the contents of a partition.
Neither does repartitioning. All these do is to rewrite the partition and
FAT tables. The data area is basically untouched until it comes to actually
writing data to it with pointers from whatever filesystem resource locator
you're using (NTFS, FAT, whatever). The only sure way of destroying data
beyond recoverability (apart from physically destroying the disk) is to
make multiple passes over hte drive with military-grade hard disk lowlevel
formatting software*.

*old Conner AT drives (of which I still have a few) had a notice on them
which said "WARNING: DO NOT LOW LEVEL FORMAT". This was all to do with the
fact that if you lowleveled the drive you had to rebuild it using the CHS
parameters for that specific unit. Get it wrong, you had a brick (or at the
very least, one which was misconfigured in such a way that you saw a
dropping sector every couple of seconds - rapidly rendering the drive
unusable). HD controllers nowadays are smart enough to rebuild themselves
after a LLF, so it's pretty safe to LLF a drive maybe half a dozen times
during its lifetime (being a very intensive operation, modern drives get
bloody hot during a LLF, so definitely not recommended without ample
cooling!)

> Thanks
>
> Stuart

--
-*- Linux Desktops & Clustering Solutions -*- http://dotware.co.uk
-*- Registered Linux user #426308 -*- http://counter.li.org
-*- Linux is like a wigwam: no Windows, no Gates, and Apache inside.
-*- <discl mode="Boilerplate" />

> What about just one pass (because it simply *is* sufficient) with freely
> available tools?

Do you make this stuff up as you go along?!

One pass is NOT sufficient to remove the data to a point at which it cannot
be recovered. It will fool most data reading tools, but certainly not
systematic, low level magnetic analysis of the platter. For that you need to
sufficiently break up the magnetic storage to the point there it is
indistinguishable from random fluctuations.

One pass might stop your basic data scavenger - it sure as hell won't stop
the pro's.

erewhon came up with this when s/he headbutted the keyboard a moment ago in
alt.computer.security:

>
>> What about just one pass (because it simply *is* sufficient) with freely
>> available tools?
>
> Do you make this stuff up as you go along?!
>
> One pass is NOT sufficient to remove the data to a point at which it
cannot
> be recovered. It will fool most data reading tools, but certainly not
> systematic, low level magnetic analysis of the platter. For that you need
to
> sufficiently break up the magnetic storage to the point there it is
> indistinguishable from random fluctuations.
>
> One pass might stop your basic data scavenger - it sure as hell won't stop
> the pro's.

hence my mention of military grade formatting tools - which destroy the data
to the point where it is practically impossible to recover anything even
with the cleanest of cleanrooms and all the time in the world - assuming
you even knew precisely what you were looking for.

I'm a pro.
--
-*- Linux Desktops & Clustering Solutions -*- http://dotware.co.uk
-*- Registered Linux user #426308 -*- http://counter.li.org
-*- Linux is like a wigwam: no Windows, no Gates, and Apache inside.
-*- <discl mode="Boilerplate" />

"Jim" <james@the-computer-shop.co.uk> wrote in message
news:B2s6h.10238$bC3.8733@newsfe7-win.ntli.net...
>> Problem is the other way around, here. I deleted a set of files, emptied
>> recycle bin, deleted more the next day and emptied again. Today the first
>> set of files were back in the folder that had been deleted from. Only
>> explanation I can think of is they there was a brief power outage that
> day,
>> so the system went down and restarted. I remember win98 and 95 used to do
>> these automatic registry restores periodically, but I didn't think that
>> deleted files and former folder contents were stored in the registry.
>>
>
> they're not - FAT file systems have two FAT tables. One's a backup; not
> that
> that does much good, because it's essentially a mirror of the primary.
> NTFS uses journaling and backgrounding to give the illusion of a faster
> filesystem. Now, the backgrounding (which makes heavy use of the large
> caches found on very modern drives) isn't much use to you if there's a
> power cut or if you're running a PVR on your system (you need realtime
> writing to disk - no caching), however the journal is where you become
> unstuck from a security viewpoint.
>
>> From a security point of view - does this mean that deleted files, with
>> recycle bin emptied, are not really deleted?
>>
>
> answer: deleting a file on an NTFS filesystem merely removes it from the
> current journal. The file is still physically on the drive. The allocated
> space is flagged for overwriting and bumped to the back of the write
> queue,
> where it is forgotten about, until it reaches the front of the write queue
> and is overwritten. On an average system, this can take /months/
> considering light usage (browsing, writing documents, etc). On a heavy-use
> system (such as a PVR) this can take a few days. Or even a few hours. Even
> then the chances of that space being entirely overwritten in order are
> fairly remote, so something of the original file will remain - very likely
> enough to use as evidence after a forensic search.

I'm ok with the forensic matters of total destruction of traces - this is
more a personal thing.
Anybody can come in and examine all of my computers all they want - there is
nothing of great importance.
But this was a collection of personal and family things for my personal
journals, and when I was finished I deleted the working copies. Nobody else
here has the ability or the interest to try to 'undelete' files. I just
expected that XP would leave them deleted, and not restore them without my
knowledge or consent.

Is there an easy way to force writing of the cache? I recall a discussion of
this in one of the linux newsgroups a few months ago.

OK, so perhaps it was just bad timing. But now all the important stuff stays
on the linux machines. Or does ext3 have the same issues?

I will test this when I have a bit of time - to see if I can duplicate the
results.

Thanks for a most useful explanation.

Stuart

Jim wrote:
> erewhon came up with this when s/he headbutted the keyboard a moment ago in
> alt.computer.security:
>
>>> What about just one pass (because it simply *is* sufficient) with freely
>>> available tools?
>> Do you make this stuff up as you go along?!
>>
>> One pass is NOT sufficient to remove the data to a point at which it
> cannot
>> be recovered. It will fool most data reading tools, but certainly not
>> systematic, low level magnetic analysis of the platter. For that you need
> to
>> sufficiently break up the magnetic storage to the point there it is
>> indistinguishable from random fluctuations.
>>
>> One pass might stop your basic data scavenger - it sure as hell won't stop
>> the pro's.
>
> hence my mention of military grade formatting tools - which destroy the data
> to the point where it is practically impossible to recover anything even
> with the cleanest of cleanrooms and all the time in the world - assuming
> you even knew precisely what you were looking for.
>
> I'm a pro.

if you're a pro then i guess you know that the military physically
destroys any media that has ever contained data above a certain security
classification because they know that nothing you do with software will
achieve the data sanitation you allude to above...

--
"it's not the right time to be sober
now the idiots have taken over
spreading like a social cancer,
is there an answer?"

"Sebastian Gottschalk" <seppi@seppig.de> wrote in message
news:4rvrh9Ftcd2cU1@mid.dfncis.de...
> Stuart Miller wrote:
>
>> Is there an easy way to force writing of the cache?
>
> Yes. 'sync' comes with every Unix, and there are various versions of
> sync.exe for Windows.
>
> But the better way is to overwrite data while bypassing the cache, because
> it doesn't fill the cache with bogus data, which would impact performance.
>

I see two convenient ways to do this

1 - open the text file, cut / delete the contents, add some garbage data,
and save
repeat cycle to drop the backup files

2 - delete file as usual, create a new file with same name in same folder,
write some garbage to it, save, exit wp, delete

Any significant difference to these approaches?

>> I recall a discussion of this in one of the linux newsgroups a few months
>> ago.
>
> Well, why don't read the manual of a good tool like SDelete for a detailed
> technical discussion of the issue?
>
Thanks - will do

It wasn't so much the paranoia of ensuring files to be unreadable, it was
just the surprise of finding them back

>> OK, so perhaps it was just bad timing. But now all the important stuff
>> stays
>> on the linux machines. Or does ext3 have the same issues?
>
> Almost all filesystems have that issue. And especially ext3, also it also
> does journaling.

Stuart