4 # Captive project doc Index page Perl template.
5 # Copyright (C) 2003 Jan Kratochvil <project-www.jankratochvil.net@jankratochvil.net>
7 # This program is free software; you can redistribute it and/or modify
8 # it under the terms of the GNU General Public License as published by
9 # the Free Software Foundation; exactly version 2 of June 1991 is required
11 # This program is distributed in the hope that it will be useful,
12 # but WITHOUT ANY WARRANTY; without even the implied warranty of
13 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
14 # GNU General Public License for more details.
16 # You should have received a copy of the GNU General Public License
17 # along with this program; if not, write to the Free Software
18 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
21 package project::captive::doc::Index;
22 require 5.6.0; # at least 'use warnings;' but we need some 5.6.0+ modules anyway
23 our $VERSION=do { my @r=(q$Revision$=~/\d+/g); sprintf "%d.".("%03d"x$#r),@r; };
28 BEGIN{ open F,"Makefile"; our $top_dir=pop @{[split /\s/,(grep /^top_srcdir/,<F>)[0]]}; eval "use lib '$top_dir'"; close F; }
34 "__PACKAGE__"=>__PACKAGE__,
35 "title"=>'Captive NTFS doc',
37 .productname { font-family: cursive; }
38 .fname { font-family: monospace; }
39 .constant { font-family: monospace; }
40 .author { font-family: cursive; }
41 .stuff { font-style: italic; font-size: larger; margin-left: 20%; margin-right: 10%; }
42 .function { font-family: monospace; }
43 .type { font-family: monospace; }
44 .command { font-family: monospace; }
45 .instruction { font-style: italic; }
53 my($img_base,$caption)=@_;
56 $r.='<table border="0" align="center">'."\n";
57 $r.="\t<tr><td>".img($img_base,$caption)."</td></tr>\n";
58 $r.="\t<caption>$caption</caption>\n";
64 sub captive_srcfile ($;$)
66 my($filename,$text)=@_;
68 a_href 'http://cvs.jankratochvil.net/viewcvs/*checkout*/priv/captive/'.$filename.'?rev=HEAD',
72 my $freespeech=a_href 'http://www.gnu.org/philosophy/free-sw.html','Free';
73 my $freebeer=a_href 'http://www.gnu.org/philosophy/free-sw.html','free (as in beer)';
79 return '<span class="productname">'.a_href($url,CGI::escapeHTML($name)).'</span>';
81 my $Wine=productname 'http://www.winehq.com/','Wine';
82 my $ReactOS=productname 'http://www.reactos.com/','ReactOS';
83 my $LinuxNTFS=productname 'http://linux-ntfs.sourceforge.net/','Linux NTFS';
84 my $GnomeVFS=productname 'http://developer.gnome.org/doc/API/gnome-vfs/','Gnome-VFS';
85 my $GnomeVFSmodule=productname 'http://developer.gnome.org/doc/API/gnome-vfs/modules.html','Gnome-VFS-module';
86 my $gnulinux='GNU/Linux';
94 <h2>Reasons for the Implementation</h2>
96 <p>Currently there is no possibility to any of the available $freespeech
97 ($freespeech used in the following text in the meaning of
98 "@{[ a_href 'http://www.gnu.org/philosophy/free-sw.html','free as in speech' ]}")
99 operating systems to reliably write to the most common disk partition
100 filesystem type – <span class="productname">Microsoft NTFS</span>. It would
101 be already supported a long time ago but there is no proper documentation of
102 <span class="productname">NTFS</span> filesystem data structures available.
103 Since <span class="productname">Microsoft</span> corporation continues in its
104 propagation of <span class="productname">Microsoft Windows NT</span>
105 (<span class="productname">NT</span> identifier used in the following text
106 applies to all the products of <span class="productname">Microsoft</span>
107 <span class="productname">NT</span> series such as
108 <span class="productname">NT 4.0</span>,
109 <span class="productname">2000</span> as NT-5.0
111 <span class="productname">XP</span> as NT-5.1.)
112 based operating systems <span class="productname">NTFS</span> is the default
113 disk file system type for vendor preinstalled <span class="productname">Microsoft Windows</span>.
115 <p>Unfortunately the <span class="productname">NTFS</span> filesystem has too
116 complex data structure to allow a complete reverse enginnering process in
117 reasonable time. Currently available $freespeech solutions such as $LinuxNTFS
118 filesystem have already implemented reliable reverse
119 engineered read-only access. However <a name="reliability">reliabile</a>
120 read-write part of the access would require much better
121 knowledge of the <span class="productname">NTFS</span> data structures.
122 Currently only rewriting of already existing file data blocks is supported
123 by $LinuxNTFS — no file creation, no file deletion, no directory operations etc.
124 Also any future versions of <span class="productname">NTFS</span> filesystem
125 would require another major reverse engineering effort.</p>
128 <h2>Challenges of the Project</h2>
130 <p>The <a name="NTFSgoal">ultimate goal</a> of this project is definitely the
131 free implementation of @{[ a_href '#reliability','reliable' ]} read-write <span
132 class="productname">NTFS</span> filesystem driver. This project chose to
133 solve this problem in the style of $Wine project by using the original binary
134 <span class="fname">ntfs.sys</span> and emulating all the required layers of
135 <span class="productname">Microsoft Windows NT</span> for it.</p>
137 <p>Unfortunately this effort is tainted by only partial and generally
138 insufficient documentation of API between filesystem driver
139 (<span class="fname">ntfs.sys</span>) and the
140 <span class="productname">Microsoft Windows NT</span>
141 ("@{[ a_href 'http://mail.gnu.org/archive/html/libtool/2000-09/msg00000.html','W32' ]}"
142 in the following text) kernel <span class="fname">ntoskrnl.exe</span>. Note
143 that this API is a different than the one being used in the $Wine project
144 since <span class="productname">Wine</span> implements only the user space
148 <h2>Microsoft Windows Versions Compatibility</h2>
150 <p>Currently this project supports only driver files of
151 <span class="productname">Microsoft Windows XP</span> (NT-5.1)
152 in the following releases:</p>
155 <li>Microsoft Windows XP Service Pack 1a Checked Build U.S.</li>
156 <li>Microsoft Windows XP Service Pack 1a Free Build U.S.</li>
157 <li>Microsoft Windows XP (No Service Pack) Checked Build U.S.</li>
158 <li>Microsoft Windows XP (No Service Pack) Free Build U.S.</li>
161 <p>The latest list of supported driver files can be found in:
162 @{[ captive_srcfile 'src/install/acquire/w32-mod-id.captivemodid.xml' ]}</p>
164 <p>Microsoft Windows NTFS filesystem driver is capable of accessing even
165 the older formats of the filesystem. Porting to Microsoft Windows 2003
166 Server is expected to be done soon. There can be also a danger of Microsoft
167 Windows upgrading NTFS disk filesystem where you would no longer be able
168 to access your NTFS disk by your original
169 <span class="productname">Microsoft Windows</span> version.
170 This upgrade does not happen as it occurs only during complete CD-ROM
171 Microsoft Windows system installation – such operation is not performed
174 <p>There is no problem technically obtaining the needed driver files of
175 <span class="productname">Microsoft Windows XP</span> as they are freely
177 @{[ a_href 'http://www.microsoft.com/WindowsXP/pro/downloads/servicepacks/sp1/checkedbuild.asp' ]}</p>
178 There may be @{[ a_href '#law','legal reasons' ]} you would not be allowed
179 to use there files if you own license to a different version of
180 <span class="productname">Microsoft Windows</span>. Legal rights will very
181 depending on your country.</p>
184 <h1>Architecture</h1>
186 <p>The principle of the
187 project lies in the glue between
188 <span class="productname">Microsoft Windows NT</span> kernel space
189 environment and $gnulinux user space process environment:</p>
191 @{[ doc_img 'arch-W32','Microsoft Windows Subsystems Architecture' ]}
192 @{[ doc_img 'arch-captive','Captive Subsystems Architecture' ]}
194 <a name="existing_emulation"><h2>Existing Emulation Projects</h2></a>
196 <p>There were two well-known $freespeech projects emulating W32 subsystems
197 to reach the compatibility with various W32 components:
198 $Wine and $ReactOS. Sad moment is that the goals of this project do not fit
199 very well into any role in those two ones. Therefore this project went
200 its own way of emulation:</p>
202 <table align="center" border="1">
204 <th>@{[ a_href '#guestosnote','Guest-OS' ]}</th>
205 <th>@{[ a_href '#hostosnote' ,'Host-OS' ]}</th>
207 <th>W32 kernel library</th>
212 <td>W32 user space</td>
213 <td><span class="fname">ntdll.dll</span></td>
217 <td><span class="constant">i386</span> hardware</td>
218 <td>W32 kernel and user space</td>
219 <td><span class="fname">ntoskrnl.exe</span></td>
221 <tr style="height: 1ex;"></tr>
223 <td>this project</td>
226 <td><span class="fname">ntoskrnl.exe</span></td>
228 <caption>Emulation Projects Characteristics</caption>
232 <a name="guestosnote"><dt>Guest-OS</dt></a>
233 <dd>@{[ a_href 'http://www.vmware.com/support/reference/common/glossary/#guestos','Guest OS' ]}:
234 An operating system that runs inside a virtual machine.</dd>
235 <a name="hostosnote" ><dt>Host OS</dt></a>
236 <dd>@{[ a_href 'http://www.vmware.com/support/reference/common/glossary/#hostos' ,'Host OS' ]}:
237 An operating system that runs on the host machine.</dd>
240 <p>While $ReactOS provides the necessary W32 kernel subsystem emulation
241 code we also need to run such @{[ a_href '#guestosnote','Guest-OS' ]} in the
242 @{[ a_href '#hostosnote','Host-OS' ]} $gnulinux. Initially it was planned to
243 extend $Wine with the W32 kernel space emulation functionality but
244 fortunately <span class="author">Steven Edwards</span> pointed to the $ReactOS
245 which better suits the needs of this project by its already implemented W32
246 kernel space emulation.</p>
248 <p>The <a name="reactos_nocare">original reasons</a> for developing
249 $ReactOS still make no sense to the author of this project. Free
250 implementation of W32 platform standalone running on the machine hardware
251 is no longer free as most od the W32 applications are usually closed source
252 and the user still looses its freedom on the application level anyway. Even
253 in the case of available free applications there still remains the
254 disadvantage of loosing the Host-OS platform availability if implemented in
255 the $Wine style. For these ideology incompatibilities not much effort was
256 made for acceptance the fixes and improvements of $ReactOS by this project.
257 Moreover new functionality is not being implemented to the $ReactOS part
258 but it is coded in Gnome style in the project specific source files
261 <p>The most serious problem of $ReactOS is its dependence on the direct
262 <span class="constant">i386</span> hardware instead of some
263 @{[ a_href '#hostosnote','Host-OS' ]} as required by the goals of this project.
264 W32 is designed to be hardware-independent using its
265 <span class="fname">hal.dll</span>. Unfortunately $ReactOS does not follow
266 this design and thus there are needed various patches and replaces of its
267 various parts and its hardware-dependent code. Despite it $ReactOS code
268 base still made a big asset for this project.</p>
270 <p class="stuff">... and @{[ a_href 'http://www.reactos.com/','ReactOS' ]} cannot run on Linux!<br />
274 <p>Some API functions are provided both by
275 <span class="fname">ntdll.dll</span> and
276 <span class="fname">ntoskrnl.exe</span> in W32.
277 <span class="author">Casper Hornstrup</span> enlightened such functions
278 calling conventions have to be differentiated as
279 <span class="fname">ntdll.dll</span> lives in the user space (low address
280 space – below <span class="constant">0x80000000</span>) and
281 <span class="fname">ntoskrnl.exe</span> in the kernel space (high address
282 space – above <span class="constant">0x80000000</span>). Although they
283 contain slightly different set of symbols (functions)
284 <span class="fname">ntdll.dll</span> still can be considered as a user
285 space interface to the kernel space implementation by
286 <span class="fname">ntoskrnl.exe</span>.</p>
288 <p>Currently there are
289 no plans to ever extend the project's crossplatformity beyond the
290 <span class="constant">i386</span> processor
291 (<span class="constant">i386</span> used here as
292 @{[ a_href 'http://www.intel.com/','Intel' ]} architecture covering 32-bit
293 processors compatible with <span class="constant">i386</span>,
294 <span class="constant">i486</span>, ...).</p>
296 <h2>API Function Implementation Choices</h2>
298 <p>During the initial point of the project development all the API
299 functions were defined as unimplemented, of course. Any call of such
300 unimplemented function is fatal and results in program termination. When we
301 need to implement any required API function we have multiple choices to do
303 @{[ a_href '#functype_pass','Direct pass to original <span class="fname">ntoskrnl.exe</span>' ]},
304 @{[ a_href '#functype_wrap','Wrap of the original <span class="fname">ntoskrnl.exe</span> function' ]},
305 @{[ a_href '#functype_native_reactos','Native implementation – $ReactOS' ]},
306 @{[ a_href '#functype_native_wine','Native implementation – $Wine' ]}
308 @{[ a_href '#functype_native_libcaptive','Native implementation – project specific' ]}.
309 <!-- a_href '#functype_undef','Undefined function' -->
311 <h2>"patched" vs. "unpatched" Libraries</h2>
313 <p>Library is called <span class="constant">patched</span> if we require
314 loading its original binary code file. Project needs to patch it to be able
315 to trap all the function entry points. The only currently
316 <span class="constant">patched</span> library of this project is
317 <span class="fname">ntoskrnl.exe</span>.</p>
319 <p>Library is called <span class="constant">unpatched</span> if no original
320 binary code is needed since all of its functions are completely emulated by
321 @{[ a_href '#functype_native','the native implementations' ]} of this project.
322 The typical <span class="constant">unpatched</span> representative is
323 <span class="fname">hal.dll</span> as it specializes on the hardware
324 dependent code and therefore it must be completely replaced by this project
325 running in the $gnulinux operating system environment. Early versions of
326 this project had also full <span class="constant">unpatched</span>
327 <a href="#native_ntoskrnl">native implementation of
328 <span class="fname">ntoskrnl.exe</span></a> but it no longer applies.</p>
330 <h2>Memory Management</h2>
332 <p>Original <span class="productname">Microsoft Windows NT</span>
333 architecture uses two address space areas – user space and kernel space.
334 User space is mapped in the range <span class="constant">0x00000000</span>
335 to <span class="constant">0x7FFFFFFF</span>, kernel space is mapped in the
336 range <span class="constant">0x80000000</span>
337 (<span class="constant">KERNEL_BASE</span> in $ReactOS sources) to
338 <span class="constant">0xFFFFFFFF</span>. All these virtual memory ranges
339 represent addresses after their MMU (Memory Management Unit) mapping, of
340 course. More discussion can be found in the
341 <a href="http://www.microsoft.com/hwdev/platform/server/PAE/PAEmem.asp">description
342 by <span class="productname">Microsoft</span></a>.</p>
344 <p>This project runs in the virtual address space used both for the UNIX
345 user space process part and for the W32 kernel space. Therefore this
346 project defines that W32 kernel runs in the whole range
347 <span class="constant">0x00000000</span> to
348 <span class="constant">0xFFFFFFFF</span> since there are no special mapping
349 assumptions about the UNIX user space process mapping. No W32 user space
350 exists in this project. Such approach also nullifies any special memory
351 moving operations between W32 kernel space and W32 user space memory areas
352 (such as <span class="function">MmSafeCopyToUser()</span>).</p>
354 <h2>Unicode Strings and Characters</h2>
356 <p>W32 platform uses 16-bit type <span class="type">wchar_t</span> while $gnulinux uses a
357 32-bit one. This can be problem during GCC (GNU C Compiler)
358 compilation of combination of native UNIX C sources (assuming 32-bit
359 GCC with 32-bit <span class="type">wchar_t</span>) and
360 $ReactOS C sources (assuming W32 compiler with 16-bit
361 <span class="type">wchar_t</span>) for literal wide strings
362 (C source file systax: <span class="command">L"wstring"</span>).
363 Possibilities to solve this issue list:</p>
367 <p>Using <span class="constant">-fshort-wchar</span> GCC option and
368 strictly differentiate between compilation of
369 <span class="productname">ReactOS</span> code and UNIX code.</p>
371 <p>pros: No source modifications needed, no runtime performance hit.</p>
373 <p>cons: No type checking if some part of code has bad compilation
374 flags, complicated way to completely split
375 <span class="productname">ReactOS</span> and UNIX code.</p>
378 <p>Wrap all <span class="productname">ReactOS</span> literal constants
379 by some conversions function call (implemented as macro
380 <span class="function">REACTOS_UCS2()</span> by this project).</p>
382 <p>pros: Any forgotten/mistaken conversions are type-checked and warned
383 during the compilation by GCC.</p>
385 <p>cons: All compiled <span class="productname">ReactOS</span> sources
386 files containing literal wide strings have to be wrapped/modified,
387 performance hit by runtime string conversions.</p>
389 <p>This solution was chosen to get the internal sanity checking
394 <h2>Supported Binary Formats</h2>
396 <p>The native W32 binary format is identified as
397 <span class="constant">PE-32</span> (Portable Executable 32-bit), such
398 files have all the usual extensions such as
399 <span class="fname">.sys</span>, <span class="fname">.exe</span>,
400 <span class="fname">.dll</span> etc. <span class="constant">PE-32</span>
401 loading support was already implemented by $ReactOS, its memory mapping
402 specifics just had to be ported to $gnulinux environment by this project.
403 This loading support does not (yet) cover importing of debug symbols from
404 W32 <span class="fname">.PDB</span> (Program DataBase) files in $gnulinux
405 ABI (Application Binary Interface) compatible way.</p>
407 <p>This project also supports transparent loading of UNIX
408 <span class="fname">.so</span> (Shared Object file) binary format. If you
409 have W32 source files for some W32 library you can try to compile it by GCC
410 to get the shared library with $gnulinux ABI compatible debug information
411 (GCC option <span class="constant">-ggdb3</span> recommended). Beware of
412 possible compilation problems as <span class="productname">Microsoft</span>
413 C code expects <span class="constant">exception</span> handling to be
414 supported by the compiler (definitely not the case of the plain C compiler
415 of GCC) — all the exception catching code should be discarded as any
416 @{[ a_href '#exception_fatal','generated exceptions are always fatal' ]} when
417 such driver is running in the scope of this project. You can use the
418 following script of this project to compile W32 filesystem source files as
419 UNIX <span class="fname">.so</span>:
420 @{[ captive_srcfile 'src/w32-mod/ext2fsd.so-build.sh' ]}</p>
422 <p>Be aware of some differences if you use
423 <span class="constant">PE-32</span> binary format file vs.
424 <span class="fname">.so</span> format file.
425 <span class="constant">PE-32</span> use the appropriate W32 specific
426 @{[ a_href '#calltype','cdecl/stdcall/fastcall call types' ]},
427 <span class="fname">.so</span> must be completely compiled in the standard
428 UNIX @{[ a_href '#calltype_cdecl','cdecl call type semantics' ]}.
429 @{[ a_href '#functype_native','Native function implementations' ]} do not need
430 to be explicitely exported by <span class="fname">captivesym</span> as they
431 are resolved automatically by the UNIX dynamic system linker. It may be
432 surprising you will have to fix all such missing symbol exports if you
433 advance during the development from the debugging
434 <span class="fname">.so</span> file for the production version of the
435 original <span class="constant">PE-32</span> binary file.</p>
437 <a name="reverse"><h2>Reverse Engineering</h2></a>
439 <p>This project has no intentions to reverse engineer and document the
440 filesystem data structures themselves since they are being encapsulated by
441 the filesystem driver. For these reasons the resources available in
442 projects such as $LinuxNTFS get out of any possible use. This project goal
443 is to provide fully compatible API interface to the rest of the W32 system
444 to persuade the filesystem driver it is running in the native
445 <span class="productname">Microsoft Windows XP</span> environment.</p>
447 <p>All the W32 filesystem drivers are running in the W32 kernel address
448 space and this area of W32 API is not much documented by
449 <span class="productname">Microsoft</span>. Some API functions are not
450 documented at all and the others are documented insufficiently for a their
451 possibly needed reimplementation from scratch. Documentation being
452 consulted primarily consists of
453 <span class="productname">@{[ a_href 'http://msdn.microsoft.com/library/default.asp?url=/library/en-us/kmarch/hh/kmarch/kmhdr_6enb.asp','MSDN (Microsoft Developer Network) Kernel-Mode Driver Architecture: Windows DDK' ]}</span>
454 documentation and also various other 3rd party documentation resources such as
455 <span class="productname">@{[ a_href 'http://www.osr.com/ntinsider/1996/cacheman.htm',
456 'The NT Cache Manager Description' ]}</span>,
457 <span class="productname">@{[ a_href 'http://www.winntmag.com/Articles/Print.cfm?ArticleID=3864',
458 'Learn About NT'."'".'s File-system Cache' ]}</span>,
459 <span class="productname">@{[ a_href 'http://www.ntfsd.org/archive/',
460 'NT File System Developers mailing list archives' ]}</span>
462 @{[ a_href 'http://www.google.com/search?q=site%3Amicrosoft.com','fulltext searches' ]}
463 through Internet from case to case.</p>
465 <p>Sometimes no sufficient documentation was found and some code behaviour
466 had to be reverse engineered directly from the binaries of
467 <span class="fname">ntoskrnl.exe</span>,
468 <span class="fname">cdfs.sys</span>,
469 <span class="fname">fastfat.sys</span>
471 <span class="fname">ntfs.sys</span>.
472 Up to now the code was disassembled by
473 <span class="productname">@{[ a_href 'http://www.simtel.net/pub/pd/29498.html','IDA Freeware' ]}</span>
475 <span class="productname">dumpbin.exe</span> of
476 <span class="productname">Microsoft Visual Studio</span>.
477 <span class="productname">dumpbin.exe</span> is fortunately able to
478 interpret debug symbols from W32 <span class="fname">.PDB</span>
479 (Program DataBase) debug information files.</p>
481 <h3><span class="productname">dumpbin.exe</span>:</p></h3>
483 <p>You should use the following options for
484 <span class="productname">dumpbin.exe</span>:</p>
486 <blockquote class="command">
487 <p>dumpbin.exe /all /rawdata:none /disasm /pdbpath:verbose FILENAME.SYS</p>
490 <p>You should see the following line in the output:</p>
492 <blockquote class="command">
493 <p>PDB file found at '.\\FILENAME.pdb'</p>
496 <a name="WinDbg"><h3><span class="productname">WinDbg</span> Windows NT kernel debugging</h3></a>
498 <p><span class="productname">WinDbg</span> is downloadable from:
499 @{[ a_href 'http://www.microsoft.com/whdc/ddk/debugging/installx86.mspx' ]}</p>
501 <p>This is (the only?) tool able to debug filesystem drivers incl.
502 <span class="fname">ntfs.sys</span>. You will need two computers running
503 <span class="productname">Microsoft Windows</span> — one computer will run
504 <span class="productname">WinDbg</span> while the other one will be
505 frozen in remote Windows NT kernel debug mode. It does not matter which
506 <span class="productname">Microsoft Windows</span> version will be run
507 on the <span class="productname">WinDbg</span> side.</p>
509 <p>The most easy way to setup two computers is to use commercial
510 <span class="productname">@{[ a_href 'http://www.vmware.com/download/workstation.html','VMware Workstation' ]}</span>
511 where you can run two virtual machines simultaneously on single PC
512 hardware and you can connect them by a virtual serial port provided by
513 <span class="productname">VMware</span>.</p>
515 <h4><span class="productname">WinDbg</span> side setup</h4>
517 @{[ doc_img 'ntdebug-vmware-windbg',
518 '<span class="productname">VMware</span> virtual serial port'
519 .' of <span class="productname">WinDbg</span> side' ]}
521 <p>You should setup <span class="productname">WinDbg</span> according
524 @{[ doc_img 'ntdebug-windbg-port','Port settings of <span class="productname">WinDbg</span>' ]}
525 @{[ doc_img 'ntdebug-windbg-sym','Symbols files location of <span class="productname">WinDbg</span>' ]}
527 <span class="constant">Symbols</span> should point to the directory where
528 reside files extracted from the symbol archive for your version of
529 <span class="productname">Microsoft Windows</span>. In the case of the
530 recommended <span class="productname">Microsoft Windows XP Service Pack 1 Checked Build</span>
532 @{[ a_href 'http://msdl.microsoft.com/download/symbols/packages/windowsxp/xpsp1sym_x86_chk.exe' ]}</p>
534 <blockquote class="command">
535 <p># Rename xpsp1sym_x86_chk.exe contents .pdb files for WinDbg<br />
536 @{[ CGI::escapeHTML(q{for i in *.pdb*;do ext="`echo $i|sed 's/^.*\.pdb\.\(.*\)$/\1/'`";if [ "$i" = "$ext" ];then echo "BAD:$i";break;fi;base="`echo $i|sed 's/\(\.pdb\)\..*$/\1/'`";echo "md $ext";echo "move /-y $i $ext\\$base";done|sort -u|sed 's/$/'`echo -ne '\r'`'/g' >/tmp/rename.bat}) ]}</p>
539 <p>The resulting <span class="command">rename.bat</span> for
540 <span class="command">xpsp1sym_x86_chk.exe</span> can be found at:
541 @{[ a_href 'xpsp1sym_x86_chk-rename.bat.zip' ]}</p>
543 <p>The resulting directory should contain at least
544 <span class="command">sys\\ntfs.pdb</span>
546 <span class="command">exe\\ntoskrnl.pdb</span>.</p>
548 <p>Your successfuly connected target (after the steps described
549 below) should look like:</p>
551 @{[ doc_img 'ntdebug-windbg-boot','Successfuly connected <span class="productname">WinDbg</span>' ]}
553 <h4>Setup of the side being kernel-debugged</h4>
555 @{[ doc_img 'ntdebug-vmware-xpdebug',
556 '<span class="productname">VMware</span> virtual serial port'
557 .' of the side being kernel-debugged' ]}
559 <p>You must use the following options in your
560 <span class="command">c:\\boot.init</span> command-line:</p>
562 <blockquote class="command">
563 <p>/debug /debugport=COM1 /baudrate=115200</p>
566 <p>After booting this <span class="command">boot.ini</span>-entry
567 should freeze at this point
568 (if no <span class="productname">WinDbg</span> is waiting in the other
569 virtual machine):</p>
571 @{[ doc_img 'ntdebug-wait','Side being kernel-debugged waiting for <span class="productname">WinDbg</span>' ]}
574 <a name="law"><h2>Laws and Licensing Conditions</h2></a>
576 <p>If you are an <span class="productname">authorized user</span> of
577 <span class="productname">Microsoft Windows NT</span> the laws in some
578 countries give you the right to fully handle the product in any way you
579 want. Therefore you can disassemble the product even in the case you had
580 to agree with the product license forbidding such disassembly as the
581 country laws override any such license agreement.</p>
583 <h3>Microsoft Service Pack</h3>
585 <p>Sometimes you may have the legal license for
586 <span class="productname">Microsoft Windows NT</span>
587 but for various technical reasons you do not have the media and/or
588 installation ready at the place of intended use of this project.</p>
590 <p>Fortunately <span class="productname">Microsoft</span> provides
591 $freebeer update packages for its
592 <span class="productname">Microsoft Windows</span> products called
593 <span class="productname">Service Packs</span>; the latest one is
594 <span class="productname">@{[ a_href 'http://www.microsoft.com/WindowsXP/pro/downloads/servicepacks/sp1/checkedbuild.asp','Microsoft Windows XP Service Pack 1a' ]}</span>.</p>
596 <p>This downloadable file contains the full versions of the essential
597 files needed for the current stage of this product:
598 <span class="fname">ntfs.sys</span>
600 <span class="fname">ntoskrnl.exe</span>.
602 <span class="fname">cdfs.sys</span> and
603 <span class="fname">fastfat.sys</span> for testing purposes.</p>
605 <p><span class="productname">Service Pack</span> also contains
606 EULA (End User License Agreement) paper disallowing any use of
607 <span class="productname">Service Pack</span> outside its original
608 intentions. According to the laws of some countries you need to be
609 <span class="productname">authorized user</span> of the
610 <span class="productname">Microsoft Windows XP</span> product to be
611 allowed to use the files contained in such
612 <span class="productname">Service Pack</span> without the bindings of its
613 EULA. Even the interpretation of such laws may vary.</p>
615 <p>It would be a breach of the law by the project author to provide
616 automatic (=hidden) functionality to download and extract the
617 <span class="productname">Service Pack</span> files. On the other hand it
618 is perfectly legal to ask user for his/her confirmation whether he/she is
619 really the <span class="productname">authorized user</span> of
620 <span class="productname">Microsoft Windows XP</span> product and
621 download/extract the <span class="productname">Service Pack</span> files
624 @{[ doc_img 'captive-install-acquire-ask','Microsoft Windows Drivers Acquire Affirmation' ]}
626 <h2>Project Architecture</h2>
628 @{[ doc_img 'dia/arch-all','Project Components Architecture' ]}
630 <p>Most of the work of this project is located in the single box called
631 "<span class="constant">libcaptive</span>" located in the center
632 of the scheme. This component implements the core W32 kernel API by
633 various methods described in this document.
634 The "<span class="constant">libcaptive</span>" box cannot be
635 further dissected as it is just an implementation of a set of
636 @{[ captive_srcfile 'src/libcaptive/ke/exports.captivesym','API functions' ]}.
637 It could be separated to several subsystems such as the
638 @{[ a_href '#cache_manager','Cache Manager' ]},
639 Memory Manager, Object Manager, Runtime Library, I/O Manager
640 etc. but they have no interesting referencing structure.</p>
642 <p>As this project is in fact just a filesystem implementation every
643 story must begin at the device file and end at the filesystem operations
644 interface. The unified suppported interfaces are
645 <span class="productname">@{[ a_href 'http://developer.gnome.org/doc/API/2.0/glib/','GLib' ]}</span>
646 (the most low level portability, data-types and utility library for Gnome)
647 <span class="type">GIOChannel</span> (for the device access) and the custom
648 <span class="constant">libcaptive</span> filesystem API. Each of these ends
649 can be connected either to some direct interface (such as the
650 <span class="constant">captive-cmdline</span> client),
651 @{[ a_href 'http://lufs.sourceforge.net/lufs/','Linux Userland File System (LUFS)' ]}
652 or as a general $GnomeVFS filter.
653 @{[ a_href 'http://lufs.sourceforge.net/lufs/','LUFS' ]} will be used in
654 most cases as it offers standard filesystem interface by Linux kernel.
656 You can also use $GnomeVFS as it offers nice filter interface on
657 the UNIX user-privileges level for transparent operation with archives and
658 network protocols. This filter interface was used by this project to turn
659 the device reference such as <span class="fname">/dev/hda3</span> or <span
660 class="fname">/dev/discs/disc0/part3</span> to the fully accessible
661 filesystem (pretending being an "archive" in the device
662 reference). This device access can be specified by $GnomeVFS URLs such as:
664 class="fname">file:///dev/hda3#captive-fastfat:/autoexec.bat</span></p>
666 <span class="constant">captive-bug-replay</span> serves just for debugging
667 purposes — you can 'replay' existing
668 <span class="fname">file.captivebug.xml.gz</span> automatically being
669 generated during W32 filesystem failure. This bugreport file will contain
670 all the touched data blocks of the device used in the moment of the
671 failure. <span class="constant">captive-bug-replay</span> will therefore
672 emulate internal virtual writable device out of these bugreported data.
674 <p>If the passed device reference is requested by the user to be accessed
675 either in <span class="dashdash">--ro</span> (read-only) mode or in the
676 <span class="dashdash">--rw</span> (full read-write) mode there are no
677 further device layers needed. Just in the case of <span
678 class="dashdash">--blind</span> mode another layer is involved to emulate
679 read-write device on top of the real read-only device by the method of
680 non-persistent memory buffering of all the possible write requests.</p>
682 <span class="constant">sandbox commit buffer</span> is involved only in the
683 case @{[ a_href '#sandbox','sandboxing feature' ]} is active. It will
684 buffer any writes to the device during the sandbox run to prevent
685 filesystem damage if the driver would fail in the meantime. If the
686 filesystem gets finally successfully unmounted this sandbox buffer can be
687 <a name="safe_flush">safely flushed</a>
688 to its underlying physical media. The buffer will be dropped
689 in the case of filesystem failure, of course. The filesystem should be
690 unmounted from time to time — it can be transparently unmounted and mounted
691 by <span class="command">commit</span> of
692 <span class="constant">captive-cmdline</span> custom client. Currently you
693 cannot force remounting when using
694 @{[ a_href 'http://lufs.sourceforge.net/lufs/','LUFS' ]} interface client
695 but it will be remounted after approx each 1MB data written automatically
696 due to @{[ a_href '#log_file_full','NTFS log file full' ]}.
698 Now we need to transparently
699 @{[ captive_srcfile 'src/libcaptive/sandbox/sandbox.idl','connect' ]}
700 the device interface of <span class="type">GIOChannel</span> type through
701 @{[ a_href '#sandbox','CORBA/ORBit' ]} to the sandboxed slave.
703 <p>Such device is still only a UNIX style GLib <span
704 class="type">GIOChannel</span> type at this point. As we need to supply it
705 to the W32 filesystem driver we must convert it to the W32 I/O Device
706 with its capability of handling <span class="type">IRP</span>
707 (<span class="constant">I/O Request Packet</span>; structure holding the
708 request and result data for any W32 filesystem or W32 block device
710 requests from its upper W32 filesystem driver. Such W32 I/O Device can
711 represent either <span class="type">CD-ROM</span> or
712 <span class="type">disk</span> device type as different W32 filesystem
713 drivers require different media types — currently only
714 <span class="fname">cdfs.sys</span> requires
715 <span class="type">CD-ROM</span> type.</p>
717 <p>W32 media I/O Device is accessed from the W32 filesystem driver.
718 The filesystem driver itself always creates volume object by
719 <span class="function">IoCreateStreamFileObject()</span> representing the
720 underlying W32 media I/O Device as the object handled by the
721 filesystem driver itself. All the client application filesystem requests
722 must be first resolved at the filesystem structures level, passed to the
723 volume stream object of the same filesystem and then finally passed to the
724 W32 media I/O Device (already implemented by this project as an
725 interface to <span class="type">GIOChannel</span> noted above).</p>
727 <p>The filesystem driver is called by the core W32 kernel implementation of
728 <span class="constant">libcaptive</span> in
729 @{[ a_href '#synchronous','synchronous way' ]} in single-shot manner instead of
730 the several reentrancies while waiting for the disk I/O completions as can
731 be seen in the original
732 <span class="productname">Microsoft Windows NT</span>.
733 This single-shot synchronous behaviour is possible since all the needed
734 resources (disk blocks etc.) can be always presented as instantly ready as
735 their acquirement is solved by @{[ a_href 'hostosnote','Host-OS' ]} outside of
736 the W32 emulated @{[ a_href 'guestosnote','Guest-OS' ]} environment.
737 For several cases needed only by <span class="fname">ntfs.sys</span>
738 there had to be supported asynchronous access — parallel execution
739 is emulated by GLib <span class="function">g_idle_add_full()</span>
740 with <span class="function">g_main_context_iteration()</span> called during
741 <span class="function">KeWaitForSingleObject()</span>.</p>
743 <p><span class="constant">libcaptive</span> offers the W32 kernel
744 filesystem API to the upper layers. This is still not the API the common
745 W32 applications are used to as they use W32 libraries which in turn pass
746 the call to W32 kernel. For example
747 <span class="function">CreateFileA()</span> is being implemented by several
748 libraries such as <span class="fname">user32.dll</span> as a relay
749 interface for the kernel function
750 <span class="function">IoCreateFile()</span> implemented by this
751 project's <span class="constant">libcaptive</span> W32 kernel
752 emulation component.</p>
754 <p>As it would be very inconvenient to use the legacy, bloated and UNIX
755 style unfriendly W32 kernel filesystem API this project offers its own
756 @{[ a_href '#client_interface','custom filesystem API interface' ]} inspired by
757 the $GnomeVFS client interface adapted to the specifics of W32 kernel API.
758 This interface is supposed to be easily utilized by
759 <a href="#client_interface_customapp">a custom application accessing
760 the W32 filesystem driver</a>.</p>
762 <p>@{[ a_href '#sandbox','CORBA/ORBit' ]} hits us again – we need to
763 @{[ captive_srcfile 'src/libcaptive/sandbox/sandbox.idl','translate' ]}
764 the @{[ a_href '#client_interface','custom filesystem API interface' ]}
765 out of the sandboxed slave to the UNIX space.</p>
767 <p><span class="constant">captive sandbox master</span> provides the
768 functionality of covering any possible sandboxed slave restarts and its
769 communication. It is also capable of
770 <a name="demultiplexing_master">demultiplexing single API operations</a>
771 to multiple its connected sandbox slaves in transparent way
772 as each of them handles
773 @{[ a_href '#mounted_one','just one filesystem device' ]}.</p>
775 <p>The rest of the story is not much special for this project since this is
776 a common UNIX problem how to offer user space implemented UNIX filesystem
777 as a generic system filesystem (as those are usually implemented only as
778 the components od UNIX kernel).</p>
780 <p>The filesystem service can be offered in several ways:</p>
783 <dt>Custom client</dt>
785 <p>One possibility would be to write
786 <a name="client_interface_customapp">a custom client application</a>
787 for this project such as file manager or a shell. Although it
788 would implement the most appropriate user interface to the set of
789 functions offered by this project (and W32 filesystem API) it has the
790 disadvantage of special client software. Appropriate client is provided
792 <span class="fname">src/client/cmdline/cmdline-captive</span></p>
795 <dt>@{[ a_href 'http://lufs.sourceforge.net/lufs/','Linux Userland File System (LUFS)' ]}</dt>
797 <p>The most usable interface is the
798 @{[ a_href 'http://lufs.sourceforge.net/lufs/','LUFS' ]} client
799 by <span class="constant">liblufs-captivefs</span>.
800 As @{[ a_href 'http://lufs.sourceforge.net/lufs/','LUFS' ]}
801 already assigns separate process for each filesystem mount the
802 @{[ a_href '#demultiplexing_master','demultiplexing feature' ]}
803 is not utilized in this case.</p>
805 <p>@{[ a_href 'http://lufs.sourceforge.net/lufs/','LUFS' ]}
806 needs multiple operating threads (each UNIX kernel operation needs
807 one free lufsd slot/thread to not to fail immediately).
808 As <span class="constant">libcaptive</span> is
809 @{[ a_href '#synchronous','single-threaded' ]} all the operations
810 get always synchronized by
811 <span class="constant">liblufs-captivefs</span>
812 before their pass over to <span class="constant">libcaptive</span>.</p>
815 <dt>@{[ a_href '#offered_gnomevfs','Gnome-VFS' ]}</dt>
817 <p>This client allowing its filesystem access even without any
818 involvement of UNIX kernel from any $GnomeVFS aware client application
819 (such as <span class="fname">gnome-vfs/tests/test-shell</span>).
820 This @{[ a_href '#offered_gnomevfs','Gnome-VFS interface' ]} connects the
821 data flow of this project in two points — both as the lowest layer
822 device image source and also as the upper layer for the filesystem
823 operation requests.</p>
827 <p>Unimplemented and deprecated methods for providing filesystem
831 <dt>W32 filesystem in UNIX OS kernel</dt>
833 <p>The real UNIX OS filesystem implementation must be completely
834 implemented inside the hosting OS kernel. This requires special coding
835 methods with limited availability of coding features and libraries.
836 Also it would give the full system control to the untrusted W32
837 filesystem driver code with possibly fatal consequences of yet
838 unhandled W32 emulation code paths. It would benefit from the best
839 execution performance but this solution was never considered a real
843 <dt>Custom NFS server</dt>
845 <p>The common approach
846 <a name="offered_NFS">of filesystem implementations</a>
847 outside UNIX OS kernel were custom NFS servers usually running on the
848 same machine as the NFS-connected client as such NFS server is usually
849 an ordinary UNIX user space process. It would be possible to implement
850 this project as a custom NFS server but the NFS protocol itself
851 has a lot of fundamental flaws and complicated code for backward
857 <a name="mounted_one"><h2>At Most One Mounted Filesystem</h2></a>
859 <p>The project technically supports only one (exactly one...) mounted
860 filesystem device and only one filesystem driver. There is nothing
861 complicated to support multiple disks and multiple loaded filesystem
862 modules but as they would share the address space it would only bring
863 a possible complications during bug reports and the bug solving
864 itself. It was considered as a more sane way to support multiple W32
865 mounted disks by completely separately running project instances in
866 a different UNIX processes communicating from their sandboxes via
867 @{[ a_href '#sandbox','CORBA sandbox interface' ]}. This sandboxing
868 feature is not yet deployed although its code is already prepared.</p>
870 <p>The project also does not support any state cleanup to be able to load
871 filesystem <span class="constant">A</span>,
872 cleanup <span class="constant">A</span> and load a different
873 filesystem <span class="constant">B</span> in the same process address
874 space. It complies with the preventions of the possible debugging
875 complications as noted above. Despite this you still must call the function
876 <span class="function">captive_shutdown()</span> to flush all the pending
877 filesystem buffers to the disk. After calling
878 <span class="function">captive_shutdown()</span> the process address space is
879 no longer usable for any further project operations and the process is
880 expected to be terminated in the manner compatible with its driving
881 @{[ a_href '#sandbox','CORBA sandbox interface' ]} control master.</p>
883 <p>Each sandbox executing the untrusted W32 binary filesystem driver code
884 is connected through its
885 @{[ a_href '#sandbox','CORBA sandbox interface' ]} at the point of upper
886 layer <span class="constant">libcaptive</span>-specific filesystem API, at
887 the point of the bottom layer of <span class="type">GIOChannel</span>
888 device access and also for transfers of GLib logging
889 messages/warnings/errors out of the sandbox to the user.</p>
892 <a name="sandbox"><h2>Sandboxing of W32 filesystem</h2></a>
894 <p>The emulated W32 environment running the original W32 filesystem driver
895 is separated from the rest of UNIX OS. It achieves the following goals:</p>
898 <li><b>Restartable</b>: W32 driver can be restartde in clean state if it crashed</li>
899 <li><b>Secure</b>: Malicious W32 code cannot affect the security of UNIX OS</li>
900 <li><b>Stable</b>: Buggy W32 cannot crash any part of UNIX OS</li>
903 <p>Sandboxing is provided with the following attributes:</p>
906 <li>standalone UNIX process with separate memory space</li>
907 <li>chroot(2) in empty directory to prevent any UNIX OS filesystem access</li>
908 <li>setuid(2) to own user/group to prevent interaction with UNIX processes</li>
909 <li>setrlimit(2) to limit system resources available for W32 environment</li>
910 <li>the only connection with the UNIX OS by CORBA/ORBit RPC</li>
913 <p>This security is almost the same as provided by
914 emulated virtual machines such as
915 @{[ a_href 'http://www.vmware.com/solutions/security.html','VMware' ]}.</p>
917 @{[ doc_img 'dia/inheritance','Sandboxing Scheme' ]}
919 <p>Project can be also used in non-sandboxed mode by
920 <span class="command">--no-sandbox</span> option as it is easier to debug
921 without CORBA/ORBit RPC. In this case the
922 <span class="type">DirectorySlave</span>/<span class="type">FileSlave</span>
923 options are used directly instead of their
924 <span class="type">DirectoryParent</span>/<span class="type">FileParent</span>
928 <h1>Choice of the Emulation Methods</h1>
930 <p>The intent of the project was to get reliable read-write access to
931 <span class="productname">NTFS</span> partition. There are several possible
932 ways to achieve that:</p>
934 <h2>Virtualmachine Running the Original W32 Subsystem</h2>
936 <p>Creating virtual-hardware PC and running the original W32 binaries
937 including their boot-loader etc. Disk device access would be passed as
938 virtual IDE disk (=hard disk drive). File access API would be implemented
939 either by special escaping by some trapped instruction out of the
940 virtualmachine while using W32 file access API or using the standard W32
941 SMB (Server Message Block) network access through some virtual network
942 card. The latter network access solution is almost the currently available
943 possibility of running full-blown disk-sharing real
944 <span class="productname">Microsoft Windows NT</span> inside virtual
945 machine emulator such as <span class="productname">VMware</span>.</p>
947 <p>pros: Full compatibility due to fully native codebase.</p>
949 <p>cons: Hard to debug, missing documentation of NT booting internals,
950 possible problems by different PC virtual-hardware than expected by NT,
951 requirement of fully installed
952 <span class="productname">Microsoft Windows NT</span> product.</p>
954 <a name="method_ntoskrnl"><h2>"ntoskrnl.exe" Inside Virtual Address Space</h2></a>
956 <p>This solution was chosen by the project. Binary filesystem driver and
957 also <span class="fname">ntoskrnl.exe</span> binary file are required.
958 Unfortunately <span class="fname">ntoskrnl.exe</span> expects a native
959 PC virtual-hardware missing during regular UNIX user space process
960 emulation, therefore such instructions must be trapped and emulated/ignored
961 from case to case.</p>
963 <p>Also the <a name="init_ntoskrnl">initialization code of <span
964 class="fname">ntoskrnl.exe</span></a> is not executed by this project since
965 it expects to get full PC hardware access privileges and thus some
966 datastructures do not get initialized by it (need to be trapped later at
967 runtime stage). Some of the missing initializations are solved by
968 @{[ a_href '#functype_wrap','API functions wrapping' ]}.
970 <p>pros: Lightweight, easier to debug.</p>
972 <p>cons: Possible incompatible emulation of
973 <span class="fname">ntoskrnl.exe</span> parts, missing documentation needed
974 for the implementation.</p>
976 <h2>Filesystem Driver Inside Virtual Address Space</h2>
978 <p>Unlike @{[ a_href '#method_ntoskrnl','previous method' ]} here we do not use
979 even <span class="fname">ntoskrnl.exe</span> as the complete kernel part of
980 W32 is <a name="native_ntoskrnl">emulated from the project source
981 files</a>. <span class="fname">cdfs.sys</span> driver was successfuly ran
982 in this manner in the former versions of this project but the possibility
983 to run without <span class="fname">ntoskrnl.exe</span> was dropped since it
984 had no licensing gains (you need the original
985 <span class="productname">Microsoft Windows NT</span> files at least for
986 the filesystem driver itself) and the emulation of undocumented parts
987 reusable from <span class="fname">ntoskrnl.exe</span> binary was
990 <p>pros: Lightweight, easier to debug.</p>
992 <p>cons: Possible incompatible emulation of the whole
993 <span class="fname">ntoskrnl.exe</span>, its missing documentation.</p>
996 <h1>Implementation Details</h1>
998 <a name="cache_manager"><h2>NT Cache Manager</h2></a>
1000 <p>Although there exist some 3rd party documents about
1001 <span class="productname">NT Cache Manager</span> W32 subsystem such as
1002 <span class="productname">@{[ a_href 'http://www.osr.com/ntinsider/1996/cacheman.htm',
1003 'The NT Cache Manager Description' ]}</span> or
1004 <span class="productname">@{[ a_href 'http://www.winntmag.com/Articles/Print.cfm?ArticleID=3864',
1005 'Learn About NT'."'".'s File-system Cache' ]}</span>
1006 they are definitely insufficient for compatible
1007 <span class="productname">NT Cache Manager</span> reimplementation.</p>
1009 <p><span class="productname">NT Cache Manager</span> is about mapping
1010 filesystem objects such as regular file data, filesystem bitmap or
1011 journalling zone (log file). It is also being used by the filesystem for
1012 mapping of virtual volume files representing the whole underlying
1013 filesystem device.</p>
1015 <p>The original W32 <span class="productname">NT Cache Manager</span>
1016 is much more complicated as it must coordinate its effort with
1017 other W32 subsystems like mapping of executable files
1018 (<span class="type">ImageSectionObject</span>), insufficient system
1019 resources from <span class="productname">NT Memory Manager</span>
1020 or general effort to perform caching features for system performance.</p>
1021 <span class="productname">NT Cache Manager</span> of this project has much
1022 simpler goal - it just needs to provide compatible
1023 <span class="productname">NT Cache Manager</span> functionality while
1024 the other goals of its W32 counterpart are left to be successfuly handled
1025 by UNIX OS in much more efficient way.</p>
1027 @{[ doc_img 'dia/cache-manager',
1028 '<span class="productname">NT Cache Manager</span> Architecture' ]}
1030 <p>Cache Manager objects are always bound to
1031 <span class="type">FCB</span> (File Control Block).
1032 <span class="type">FileObject</span> (or its associated
1033 <span class="type">HANDLE</span>) serve only as reference
1034 to <span class="type">FCB</span> and there can be multiple
1035 <span class="type">FileObject</span>/<span class="type">HANDLE</span>
1036 items for one <span class="type">FCB</span>. It is a bit misleading
1037 you must use <span class="type">FileObject</span> pointer while calling
1038 most of the Cache Manager functions.</p>
1040 <p>Before using any other Cache Manager functions you must first call
1041 <span class="function">CcInitializeCacheMap()</span>. You must give the
1042 maximum mapped object offset. Each mapped object byte must have at most one
1043 mapped memory location - no shared pages are allowed. Also any subsequent
1044 mapping request is expected to be mapped into continuous memory region.
1045 It implies you must reserve the memory region for possible future mapping
1046 during the initial <span class="function">CcInitializeCacheMap()</span>
1047 moment sized according to the given maximum mapped object offset.
1048 This is the approach currently implemented by this project although it
1049 cannot be used for 3rd party <span class="fname">ext2fsd.sys</span>
1050 driver as it initialized Cache Manager by the whole media device size
1051 and it surprisingly succeeds for original
1052 <span class="productname">Microsoft Windows</span>
1053 <span class="productname">Cache Manager</span>.
1054 I expect the space reservation should be postponed to the first mapping
1055 request and expect no multiple mappings will be done in the case
1056 of memory-exceeding <span class="function">CcInitializeCacheMap()</span>
1057 reservation request. <span class="function">CcSetFileSizes()</span>
1058 changing the reserved memory area size may assume no existing Map
1059 or Pin mappings exist.</p>
1061 <p><span class="type">PCACHE_MANAGER_CALLBACKS</type> argument can be
1065 <dt><span class="function">AcquireForReadAhead()</span>/<span class="function">ReleaseFromReadAhead()</span></dt>
1067 <p>As any readahead functionality is optional these entries are
1068 never used by Cache Manager implementation of this project.</p>
1071 <dt><span class="function">AcquireForLazyWrite()</span>/<span class="function">ReleaseFromLazyWrite()</span></dt>
1073 <p>Even the write-behind functionality is optional for Cache Manager.
1074 It is being done in asynchronous way in the original
1075 <span class="productname">Microsoft Windows</span>
1076 <span class="productname">Cache Manager</span>.
1077 implementation and it is ignored by Cache Manager implementation of
1080 <p>Cache Manager does not need to write any data if not explicitely
1081 requested by the driver. It is even expected to silently drop any
1082 pending dirty data blocks during filesystem shutdown.
1083 Forced dirty block write by function
1084 <span class="function">CcFlushCache()</span> should be written without
1085 any wrapping surrounding
1086 <span class="function">AcquireForLazyWrite()</span>/<span class="function">ReleaseFromLazyWrite()</span>
1091 <p><span class="function">CcUninitializeCacheMap()</span> is just
1092 a suggestion for Cache Manager that driver will no longer reference
1093 given <span class="type">SharedCacheMap</span>. The uninitialization
1094 can be postponed to any later moment in original
1095 <span class="productname">Microsoft Windows</span>
1096 <span class="productname">Cache Manager</span>
1097 as it may be locked by existing
1098 <span class="type">ImageSectionObject</span>
1099 of some file being executed etc.
1100 <a name="sharedcachemap_leak">It is fatal to destroy
1101 <span class="type">SharedCacheMap</span></a>
1102 in the moment you see no other
1103 references to it as the driver will access it for some moment
1104 even after <span class="function">CcUninitializeCacheMap()</span>.
1105 I am not sure if it is a bug of the driver or whether there are some rules
1106 how long after <span class="function">CcUninitializeCacheMap()</span>
1107 completion given <span class="type">SharedCacheMap</span> still exists.
1108 Fortunately it is safe to never destroy
1109 <span class="type">SharedCacheMap</span> and leave it leaked - everything
1111 @{[ a_href '#sandbox','sandboxed environment' ]} soon anyway.</p>
1113 <p>There exist Map and Pin type objects for each
1114 <span class="type">SharedCacheMap</span> although they look very similiar.
1115 Only these objects give you access to any memory data
1116 — <span class="type">SharedCacheMap</span> only reserved the space
1117 to ensure continuous mapping of the forthcoming mappings but it did not map
1118 any data into it.</p>
1120 <p>Mapping of 'new' Map or Pin will create the new object only in the case
1121 no such mapping exists now. Otherwise you will just get the reference to
1122 the existing object with increased usecount.</p>
1127 <p>Map mapping is always at most one for each
1128 <span class="type">SharedCacheMap</span>. Base offset/length of such
1129 mapping have no meaning as there can be only single Map.</p>
1131 <p>Apparently Map size can be arbitrary long according
1132 to its <span class="type">SharedCacheMap</span> reserved space.</p>
1134 <p>You cannot modify the memory mapped by Map in any way.
1135 As it is the same memory area (address) as the pages used by Pin
1136 objects you always access the last modified version by possible
1137 Pin of the same page.</p>
1142 <p>Pin mapping always represents just one physical page
1143 (<span class="constant">PAGE_SIZE</span> &nspan; 4096 for i386).
1144 Its base offset/length can be safely extended to be aligned to the
1147 <p>Pin can have associated pair of oldest and newest
1148 <span class="type>LSN</span> (Linear Sequence Number). It can be
1149 set by <span class="function">CcSetDirtyPinnedData()</span>
1150 and Cache Manager always tracks the lowest and highest
1151 reported <span class="type>LSN</span> for each page.
1152 <span class="type>LSN</span> is assumed to be
1153 <span class="constant">0</span> if not set.</p>
1155 <p>Any existing Pin mapping will be reused for further mappings
1156 as long as it is not ThreadOwned. In the moment you use
1157 <span class="function">CcSetBcbOwnerPointer()</span> you will detach
1158 the associated Pin pages from its
1159 <span class="type">SharedCacheMap</span>.
1160 Although they will further act as valid Pin mappings they will be no
1161 longer reused during new Pin mapping of the same page.
1162 There can exist multiple Pin mappings of the same page (although
1163 sharing the same memory space). This detaching must be implemented
1165 @{[ a_href '#synchronous','single-threaded' ]} W32 implementation
1166 of this project as it is affecting the behaviour of Cache Manager.
1168 @{[ a_href '#TraceFS','seen' ]} how to behave if multiple dirty Pin
1169 mappings of the same page exist.</p>
1173 <p>Only the pages not yet present in the memory must be read from the disk.
1174 You must not read any pages you do not need to as the driver does not
1175 expect it and it would corrupt its data buffers.</p>
1177 <p>Cache Manager of this project will destroy Pin or Map mappings after
1178 their last unreferencing (in opposite of
1179 @{[ a_href 'sharedcachemap_leak','leaked <span class="type">SharedCacheMap</span>' ]}).
1180 Despite it any dirty pages may still be held as the pages
1181 (including their <span class="type>LSN</span>s) are cached associated
1182 with <span class="type">SharedCacheMap</span>. It may be also possible
1183 original <span class="productname">Microsoft Windows</span>
1184 <span class="productname">Cache Manager</span>
1185 postpones Pin mapping destroy to later time but it does not matter.</p>
1188 <h3>TraceFS NT Cache Manager Tracer</h3>
1190 <p>@{[ a_href '#cache_manager','Cache Manager behaviour' ]} would be hard
1191 to analyze just by @{[ a_href '#reverse','reverse engineering' ]} as it
1192 is pretty complicated code cooperating with many other W32 kernel
1193 subsystems. It was chosen as easier way to trace it instead and validate
1194 all the Cache Manager assumptions by Cache Manager simulator.</p>
1196 @{[ doc_img 'dia/TraceFS','TraceFS Hooking' ]}
1198 <p>You must prepare your driver to be hooked
1199 (<span class="fname">ntfs.sys</span> in this case):</p>
1201 <blockquote class="command">
1202 <p>@{[ captive_srcfile './src/TraceFS/hookfs.pl' ]} ntfs.sys ./src/TraceFS/TraceFS-W32/TraceFS.sys >hooked/ntfs.sys</p>
1205 <p>This <span class="fname">hooked/ntfs.sys</span> file must be replaced
1206 in the <span class="fname">%System32%\\drivers</span> directory.
1208 <span class="productname">Microsoft Windows</span>
1209 has many backups of these system files such as
1210 <span class="fname">%System32%\\dllcache</span> — delete them
1213 <p>You also need to install
1214 <span class="fname">./src/TraceFS/TraceFS-W32/TraceFS.sys</span>
1215 into <span class="fname">%System32%\\drivers</span> directory
1216 and import <span class="fname">TraceFS/TraceFS-W32/TraceFS.reg</span>
1217 registry file to initialize the debug driver during system boot.</p>
1219 <p>You can now pray a bit and snap the resulting Cache Manager tracing
1220 from <span class="productname">WinDbg</span> by
1221 @{[ a_href '#WinDbg','W32 remote kernel debugging' ]}:</p>
1223 @{[ doc_img 'ntdebug-windbg-boot','Successfuly connected <span class="productname">WinDbg</span>' ]}
1225 <p>The resulting trace file should be processed by
1226 @{[ captive_srcfile './src/TraceFS/checktrace.pl' ]} Perl Cache Manager
1227 implementation to validate its assumptions about Cache Manager behaviour.
1228 Any seen incompatibilies will be reported — your target is to reach
1229 as few error messages as possible.</p>
1232 <h4>TraceFS for general API tracing</h4>
1234 <p>Although TraceFS was up to now used only for tracing of
1235 <span class="productname">NT Cache Manager</span> it can be easily
1236 used ever for any other NT kernel API tracing. You need to provide
1237 appropriate function wrappers in the main source file
1238 @{[ captive_srcfile './src/TraceFS/TraceFS-W32/TraceFS.c' ]}
1239 and you must also export them in
1240 @{[ captive_srcfile './src/TraceFS/TraceFS-W32/TraceFS.def' ]}.
1241 @{[ captive_srcfile './src/TraceFS/hookfs.pl' ]} has no hardcoded
1242 function names – it will hook exactly the exported entries.</p>
1244 <p>Framework for thread synchronizations and debug tracing is provided to
1245 prevent mangling of messages while running by multiple threads at once.
1246 Testing was done just on uniprocessor machine, SMP kernel may need some
1250 <a name="functype"><h2>API Function Implementation Choices</h2></a>
1252 <p>For each function exported by W32
1253 <span class="fname">ntoskrnl.exe</span> and imported and called by the
1254 filesystem driver a decision needs to be made to properly implement its
1255 functionality. Currently implemented functionality statistics are provided
1258 <table border="1" align="center">
1259 <tr><th>Function type </th><th>Items</th><th>Portion</th></tr>
1260 <tr><td>@{[ a_href '#functype_pass','pass' ]} </td><td> 81</td><td> 26%</td></tr>
1261 <tr><td>@{[ a_href '#functype_wrap','wrap' ]} </td><td> 2</td><td> 0%</td></tr>
1262 <tr><td>@{[ a_href '#functype_native_reactos','native-ReactOS' ]}</td><td> 113</td><td> 36%</td></tr>
1263 <tr><td>@{[ a_href '#functype_native_libcaptive','native-own' ]} </td><td> 116</td><td> 38%</td></tr>
1264 <caption>Function Implementation Types Statistics</caption>
1267 <p>As there are several choices to implement each function the usual
1268 attempts/investigations ordering is listed in the sections below.</p>
1270 <p>Special case must be taken for data-type symbols since they are
1271 referenced without the possibility of catching the code flow by some
1272 breakpoints (it would be possible only in some special access cases). Data
1273 export symbols of <span class="constant">unpatched</span> libraries must
1274 contain already prepared content at the runtime. There is a problem
1275 with <span class="constant">patched</span> libraries where it is necessary
1276 to also fully implement the data symbol as
1277 @{[ a_href '#functype_native','native implementation' ]} since there is no
1278 possibility to @{[ a_href '#functype_pass','pass' ]} the data symbol instead of
1279 the original W32 data location and therefore there will be two instances of
1280 such data variable place. As there will be also the uncaught references for
1281 such W32 data location from the <span class="constant">patched</span>
1282 library itself such symbols should be usually only some constants (such as
1283 <span class="constant">KeNumberProcessors</span>).</p>
1285 <p>W32 platform symbols export/import can be based either on the symbol
1286 name itself or it can be also exported and imported just by its
1287 identification number called <span class="constant">Ordinal</span>.
1288 Although it saves some jumptables file binary size it is currently no
1289 longer used by W32 binaries and this project also does not support such
1290 <span class="constant">Ordinal</span> symbol reference type at all.</p>
1292 <p>All the exporting magic is handled by custom script
1293 <span class="fname">captivesym</span> processing the definition file
1294 <span class="fname">@{[ captive_srcfile 'src/libcaptive/ke/exports.captivesym' ]}</span>
1295 to produce the intermediate relaying code
1296 <span class="fname">src/libcaptive/ke/exports.c</span>. For details of the
1297 <span class="fname">captivesym</span>-specific source file syntax please
1298 see its documentation:
1299 <span class="fname">@{[ a_href
1300 $W->{"top_dir"}.'/project/Pod2Html.html.pl?cvs=priv/captive/src/libcaptive/ke/captivesym.pl',
1301 'src/libcaptive/ke/captivesym.pl' ]}</span>
1303 <a name="functype_pass"><h3>Direct Pass to Original "ntoskrnl.exe"</h3></a>
1305 <p>Simple (standalone) functions such as
1306 <span class="function">RtlTimeToSecondsSince1970()</span> can be simply
1307 passed to the original implementation in
1308 <span class="fname">ntoskrnl.exe</span> as they make no hardware access
1309 and they do not expect any special internal data structures to be set up
1310 in advance by an earlier library initialization. A common case are all
1311 the data structures utility functions such as
1312 <span class="constant">GenericTable</span> subsystem or
1313 <span class="constant">LargeMcb</span> handling.</p>
1315 <a name="functype_pass_fromunix"><h4>Pass from UNIX Code</h4></a>
1317 <p>Control flow begins in some standard UNIX code. Such code is always
1318 using @{[ a_href '#calltype_cdecl','cdecl call type' ]} for all its
1319 intracalls. <a href="#functype_native_reactos">Native functions
1320 compiled from <span class="productname">ReactOS</span> sources</a> use
1321 their own @{[ a_href '#calltype','cdecl/stdcall/fastcall' ]} declarations
1322 but these call type modifications are discarded during compilation for
1323 this project by the <span class="constant">LIBCAPTIVE</span>
1326 <p>UNIX code calls <span class="function">FUNCTIONNAME()</span> relay
1327 from the generated UNIX jump table. Such relay will debug dump the
1328 passed arguments and finally pass the control to the original W32
1329 function code in the proper call type
1330 @{[ a_href '#calltype','cdecl/stdcall/fastcall' ]} for a given
1333 <p>Original W32 code entry point is always trapped by a breakpoint
1334 although it would not be needed during this specific direct pass from
1335 UNIX code to the original W32 implementation. Still the breakpoint has
1336 to be there to catch some other (such as intra-W32) possible calls
1337 described later. There are several more ways to define breakpoint in
1338 the code. One way is to use processor hardware breakpoint support but
1339 the number of breakpoints is limited. The other way is to patch in the
1340 <span class="instruction">@{[ 'int $3' ]}</span> instruction but it will invoke
1341 <span class="constant">SIGTRAP</span> signal handler conflicting with
1342 the possible debugger (<span class="productname">gdb(1)</span>)
1343 control. This project uses the <span class="instruction">hlt</span>
1344 instruction, which also has a single-byte opcode as
1345 <span class="instruction">@{[ 'int $3' ]}</span> and it is a privileged
1346 instruction forbidden to be used from the UNIX user space code.
1347 <span class="instruction">hlt</span> invokes
1348 <span class="constant">SIGSEGV</span> signal which can be resolved by
1349 a custom signal handler without any conflict with the possible
1350 debugger control; <span class="productname">gdb(1)</span> needs the
1351 following command to pass through such
1352 <span class="constant">SIGSEGV</span> signal:</p>
1354 <blockquote class="command">
1355 <p>handle SIGSEGV nostop noprint pass</p>
1358 <p>When a breakpoint gets caught, we usually need to return to the
1359 running code. Unfortunately it is not possible because of the patched
1360 breakpoint opcode. The breakpoint cannot be simply removed upon return
1361 as it would permanently loose control over the point of entry. Even if
1362 the return would include faking of the return address in the bottom
1363 stack frame to patch the breakpoint back during later function exit it
1364 still would not solve the caughts of inner calls of recursive
1365 functions. One of the working possibilities would be to patch the
1366 original instruction back and perform a singlestep provided by
1367 <span class="function">ptrace(2)</span> syscall. However such
1368 singlestep needs another controlling UNIX process and it would again
1369 conflict with the debuggers such as
1370 <span class="productname">gdb(1)</span>. This project implements the
1371 singlestep functionality by two consecutive breakpoints
1372 (<span class="instruction">hlt</span> instructions to be specific):
1373 The first two instruction addresses of the W32 functions are called
1374 <span class="productname">slot #1</span> and
1375 <span class="productname">slot #2</span>, the length of the first
1376 function instruction has to be analyzed to get the right address of
1377 <span class="productname">slot #2</span>. When the first breakpoint is
1378 caught it is necessary to patch the original instruction back and also
1379 patch another breakpoint in place of
1380 <span class="productname">slot #2</span>.
1381 During the <span class="productname">slot #2</span> breakpoint
1382 invocation the operation will be reverted — the breakpoint will be put
1383 to <span class="productname">slot #1</span> again and the instruction
1384 of <span class="productname">slot #2</span> will be restored to be able
1385 to continue the execution of the function.</p>
1387 <p>W32 function will finish in its specific
1388 @{[ a_href '#calltype','cdecl/stdcall/fastcall call type' ]}, the control
1389 will return to the UNIX jump table relay which will debug dump the
1390 return value and it will finally pass the control back to the UNIX
1391 caller in the standard UNIX
1392 @{[ a_href '#calltype_cdecl','cdecl call type' ]}.</p>
1394 @{[ doc_img 'fig/functype_patched_pass_fromunix',
1395 'Function Type: <span class="constant">pass</span> from UNIX Code' ]}
1397 <a name="functype_pass_fromw32"><h4>Pass from W32 Code</h4></a>
1399 <p>This function type is similiar to the
1400 @{[ a_href '#functype_pass_fromunix','previous one' ]} with the exception
1401 of more complicated entry point. Unfortunately W32 libraries call their
1402 own functions directly, using the <span class="instruction">call</span>
1403 instructions without any patchable jump table. Even the
1404 <span class="instruction">call</span> argument itself cannot be patched
1405 according to the relocation table record as such library intra-call
1406 instruction has no relocation due to its relative argument offset on
1407 <span class="constant">i386</span>. This time the double-breakpoint
1408 mechanism @{[ a_href '#functype_pass_fromunix','described above' ]} gets
1409 handy since it will catch the entry point when the function gets
1410 called. <span class="constant">SIGSEGV</span> handler gets invoked by
1411 the <span class="instruction">hlt</span> instruction and it will
1412 redirect the control to the jump table relay function to debug dump the
1413 function entry arguments (it has no other uses in this call type).</p>
1415 <p>When the relay needs to call the original function it will reach
1416 exactly the same breakpoint instruction as during the recent
1417 <span class="constant">SIGSEGV</span> handling redirecting to this
1418 calling relay. But this time the
1419 <span class="constant">through_w32_func</span> field of this function
1420 record will be set to to prevent repeated redirection and to pass the
1421 control through the breakpoint mangle instead this time.</p>
1423 <p>Returning is not much interesting as the first
1424 <span class="constant">SIGSEGV</span> handler did a straight jump
1425 for the redirection purposes without any needed consequent
1428 <p>The jump table relay used for the callers from W32 code is
1429 a different one than the relay being used for the callers
1430 @{[ a_href '#functype_pass_fromunix','from UNIX code' ]}. UNIX code always
1431 uses relay with external @{[ a_href '#calltype_cdecl','cdecl call type' ]}
1432 but in this case a relay with the appropriate
1433 @{[ a_href '#calltype','cdecl/stdcall/fastcall call type' ]} is used.</p>
1435 @{[ doc_img 'fig/functype_patched_pass_fromw32',
1436 'Function Type: <span class="constant">pass</span> from W32 Code' ]}
1440 <table border="1" align="center">
1441 <tr><td><span class="fname">captivesym</span> keyword</td><td>pass</td></tr>
1442 <tr><td>Native code function name </td><td>(no implementation)</td></tr>
1443 <tr><td>W32 traced code from UNIX function name </td><td>FUNCNAME</td></tr>
1444 <tr><td>W32 traced code from W32 function name </td><td>FUNCNAME_cdecl/_stdcall/_fastcall</td></tr>
1445 <tr><td>Entry/exit debug tracing from UNIX code </td><td>yes</td></tr>
1446 <tr><td>Entry/exit debug tracing from W32 code </td><td>yes</td></tr>
1447 <caption>Function Type <span class="constant">pass</span> Characteristics</caption>
1450 <a name="functype_wrap"><h3>Wrap of the Original "ntoskrnl.exe" Function</h3></a>
1452 <a name="functype_wrap_fromunix"><h4>Wrapping of Call from UNIX Code</h4></a>
1454 <p>The code control flow has no special hardcore features since it is
1455 very similiar to <a href="#functype_pass_fromunix">the direct pass to
1456 W32 function from UNIX code</a>. All the wrapping is done in the
1457 standard UNIX @{[ a_href '#calltype_cdecl','cdecl call type' ]} manner.
1458 Jump table debug dumping relays are provided twice — the
1459 "outer" one to trace the parameters from the function caller
1460 and the "inner" one to trace the call from the wrapper to the
1461 original W32 code. The "inner" relay also calls the W32 code
1462 with the appropriate <a href="#calltype">cdecl/stdcall/fastcall call
1465 @{[ doc_img 'fig/functype_patched_wrap_fromunix',
1466 'Function Type: <span class="constant">wrap</span> from UNIX Code' ]}
1468 <a name="functype_wrap_fromw32"><h4>Wrapping of Call from W32 Code</h4></a>
1470 <p>This scheme is a combination of the
1471 <a href="#functype_wrap_fromunix">previous wrap of a call from
1472 UNIX code</a> and the <a href="#functype_pass_fromw32">direct pass from
1473 the W32 code</a>. The control is caught and redirected by
1474 <span class="constant">SIGSEGV</span> handler from the breakpoint
1475 placed at the entry to the original W32 function code. The second entry
1476 to the original W32 function with the
1477 <span class="constant">through_w32_func</span> field of this function
1478 description already set is done from the "inner" jump table
1479 relay with the appropriate
1480 @{[ a_href '#calltype','cdecl/stdcall/fastcall call type' ]}.</p>
1482 @{[ doc_img 'fig/functype_patched_wrap_fromw32',
1483 'Function Type: <span class="constant">wrap</span> from W32 Code' ]}
1487 <p>Some functions can be <a href="#functype_pass">passed to the original
1488 code</a> but they need their parameters to be checked/prepared.
1489 Currently, such wrapping is only needed for the
1490 <span class="function">ExAllocateFromPagedLookasideList()</span> function
1491 where it is required due to <a href="#init_ntoskrnl">missing execution of
1492 <span class="fname">ntoskrnl.exe</span> initialization execution</a>,
1493 which would otherwise properly initialize some internal data structures.
1494 In this case the wrapping code detects passing of an uninitialized
1495 parameter and will search through the whole
1496 <span class="fname">ntoskrnl.exe</span> code body at runtime to find the
1497 proper initialization routine containing the correct initialization
1498 parameters. Passed addresses of static structures must be differentiated
1499 as each of them usually has different initialization parameters. It is
1500 proactive to not to have fixed parameters array as these parameters may
1501 differ across different <span class="fname">ntoskrnl.exe</span>
1504 <table border="1" align="center">
1505 <tr><td><span class="fname">captivesym</span> keyword</td><td>wrap</td></tr>
1506 <tr><td>Native UNIX wrapping code function name </td><td>FUNCNAME_wrap</td></tr>
1507 <tr><td>W32 traced wraping code from UNIX func. name </td><td>FUNCNAME</td></tr>
1508 <tr><td>W32 traced wrapping code from W32 func. name </td><td>FUNCNAME_cdecl/_stdcall/...</td></tr>
1509 <tr><td>W32 traced original code function name </td><td>FUNCNAME_orig</td></tr>
1510 <tr><td>Entry/exit debug tracing from UNIX code </td><td>yes</td></tr>
1511 <tr><td>Entry/exit debug tracing from W32 code </td><td>yes</td></tr>
1512 <caption>Function Type <span class="constant">wrap</span> Characteristics</caption>
1515 <a name="functype_native"><h3>Native Implementation</h3></a>
1517 <h4>Native Implementation Called from UNIX Code</h4>
1519 <p>This is the simplest case of a function call as it is fully
1520 handled only by the compiler and/or linker.</p>
1522 <p>In this case though, no debug dumping call relay is provided — such
1523 relay would need to rename the implementations of native functions to
1524 prevent its automatic linking with the caller code. This renaming would
1525 not be possible to do by simple <span class="constant">#define</span>
1526 since it would also rename any calling statements of such function in
1527 the same C sources. One of the possibilities to solve would be to
1528 utilize <span class="dashdash">--redefine-sym</span> feature of the
1529 <span class="productname">objcopy(1)</span> utility. On the other hand
1530 there is not much need to catch/debug such calls as both the caller and
1531 the callee are provided with full source file debug information for the
1532 debugger. Also the callee usually debug dumps its entry/exit parameters
1533 by custom debug dumps in the
1534 <a href="#functype_native_reactos"><span class="productname">ReactOS</span> implementations</a>.
1536 @{[ doc_img 'fig/functype_native_fromunix',
1537 'Function Type: <span class="constant">native</span> from UNIX Code' ]}
1539 <a name="functype_native_fromw32"><h4>Native Implementation of
1540 "unpatched" Library Function Called from W32 Code</h4></a>
1542 @{[ doc_img 'fig/functype_unpatched_native_fromw32',
1543 'Function Type: <span class="constant">native</span> of <span class="constant">unpatched</span> from W32 Code' ]}
1545 <p>Here comes the differentiation if the project deals either with
1546 a <span class="constant">patched</span> or an
1547 <span class="constant">unpatched</span> version of the library
1548 (<span class="constant">patched</span> is a loaded W32 binary
1549 library while <span class="constant">unpatched</span> library is
1550 completely provided by this project with no use of the library's
1551 original W32 binary file). As the project adjusts the exported symbol
1552 address during the patching operation, in some cases the
1553 <span class="constant">patched</span> library call may be handled
1554 simply as <span class="constant">unpatched</span> library call even for
1555 the <span class="constant">patched</span> libraries. Fortunately the
1556 distinction is not much important as the project is prepared to
1557 properly handle both cases.</p>
1559 <p>The W32 caller which imported the symbol will be pointed right to
1560 the relaying function. The debug dumping relay will be called from W32
1561 code with the appropriate
1562 @{[ a_href '#calltype','cdecl/stdcall/fastcall call type' ]} while the
1563 relay will call the implementation of the native function in the
1564 standard UNIX @{[ a_href '#calltype_cdecl','cdecl call type' ]} manner.</p>
1566 <h4>Native Implementation of "patched" Library Function Called from W32 Code</h4>
1568 @{[ doc_img 'fig/functype_patched_native_fromw32',
1569 'Function Type: <span class="constant">native</span> of <span class="constant">patched</span> from W32 Code' ]}
1571 <p>The calling scheme is similiar to the
1572 <a href="#functype_native_fromw32">previous call of
1573 <span class="constant">unpatched</span> library function from W32
1574 code</a> but the call control is redirected from the entry point of the
1575 original W32 binary implementation by the breakpoint and its
1576 <span class="constant">SIGSEGV</span> handler as in
1577 <a href="#functype_pass_fromw32">the case of passing control from W32
1580 <p>The original W32 function implementation located in the original
1581 loaded binary file is never executed but its entry point needs to be
1582 trapped by the breakpoint to be able to catch the function calls within
1587 <p>In all cases the final function implementation is a standard UNIX
1588 code compiled from C sources with full debug information available
1589 for the debugger. Fortunately all such functions do not need to be coded
1590 from scratch for this project since there already exist $freespeech
1591 $ReactOS and $Wine projects and their code can be used instead.</p>
1593 <p>$Wine project is listed mostly for a completeness as almost no
1594 code was suitable for reuse as it implements W32 user space while this
1595 project is running pure W32 kernel space environment (in $gnulinux user
1598 <a name="functype_native_reactos"><h4>Native Implementation
1599 - <span class="productname">ReactOS</span></h4></a>
1601 <p>Some functions are already implemented in the $ReactOS
1602 project and they can be used as they are. Although it would be
1603 possible to <a href="#functype_pass">pass some function calls to the
1604 original code</a> it is more handy to provide native implementation as
1605 there is better control of the data handling during debugging sessions
1606 due to the provided debugging symbols.</p>
1608 <p>Such functions can be found in
1609 <span class="fname">src/libcaptive/reactos/</span> subdirectory.
1610 Some functions had to be adjusted for this project
1611 - these modifications are compiled conditionally, depending on the
1612 <span class="constant">LIBCAPTIVE</span> symbol existence.</p>
1614 <p>Later stages of this project reached the level where
1615 $ReactOS is yet too immature and the needed functions are usually
1616 written just with the sad body:</p>
1618 <blockquote class="command">
1619 <p>UNIMPLEMENTED;</p>
1622 <p>Functions that were not possible to
1623 @{[ a_href '#functype_pass','pass' ]} were reimplemented by this project
1624 and placed in the project's implementation directories
1625 @{[ a_href '#reactos_nocare','instead of extending' ]} $ReactOS code.</p>
1627 <a name="functype_native_wine"><h4>Native Implementation – <span class="productname">Wine</span></h4></a>
1629 <p>Even though $Wine only implements the
1630 <span class="productname">Microsoft Windows NT</span> user space, there
1631 still are some common functions which could be copied from the $Wine
1634 <a name="functype_native_libcaptive"><h4>Native Implementation – Project Specific</h4></a>
1636 <p>As the last resort it was necessary to provide completely own
1637 implementation of some API functions such as PC hardware dependent
1638 parts or memory management functions.</p>
1642 <table border="1" align="center">
1643 <tr><td><span class="fname">captivesym</span> keyword</td><td>(none; just the symbol name)</td></tr>
1644 <tr><td>Native code function name </td><td>FUNCTIONNAME</td></tr>
1645 <tr><td>Native traced code from W32 code func. name </td><td>FUNCTIONNAME_cdecl/_std...</td></tr>
1646 <tr><td>Entry/exit debug tracing from UNIX code </td><td>no</td></tr>
1647 <tr><td>Entry/exit debug tracing from W32 code </td><td>yes</td></tr>
1648 <caption>Function Type <span class="constant">native</span> Characteristics</caption>
1651 <a name="functype_undef"><h3>Undefined Function</h3></a>
1653 <p>Functions not defined by any of the previous function types cannot be
1654 called by any W32 code including the code of the library implementing
1655 such function. All functions of <span class="constant">patch</span>ed
1656 libraries not listed in the <span class="fname">captivesym</span> exports
1657 file are automatically set to be trapped as fatal program execution
1660 <p>It is not necessary to list the symbols as
1661 <span class="constant">undef</span> as long as you are just loading the
1662 W32 <span class="constant">PE-32</span> code and the symbols belong to
1663 <span class="constant">patch</span>ed library. On the other hand if you
1664 are loading W32 <span class="fname">.so</span> code or if such symbol is
1665 a part of <span class="constant">unpatched</span> library (and thus
1666 being completely provided by the project) you need to list such symbol as
1667 <span class="constant">undef</span> type to prevent unresolved symbol
1670 <table border="1" align="center">
1671 <tr><td><span class="fname">captivesym</span> keyword</td><td>undef</td></tr>
1672 <tr><td>Native code function name </td><td>(no implementation)</td></tr>
1673 <tr><td>Native traced code function name </td><td>FUNCTIONNAME_cdecl/_stdcall/_fastcall</td></tr>
1674 <tr><td>Debug tracing message from UNIX code </td><td>yes</td></tr>
1675 <tr><td>Debug tracing message from W32 code </td><td>yes</td></tr>
1676 <caption>Function Type <span class="constant">undef</span> Characteristics</caption>
1680 <a name="calltype"><h2>API Function Calling Conventions</h2></a>
1682 <p>Standard UNIX code compiled by GCC (GNU C Compiler) running on host
1683 $gnulinux always uses @{[ a_href '#calltype_cdecl','cdecl' ]} ABI (Application
1684 Binary Interface) calling convention. This calling convention is also the
1685 default declaration type of UNIX functions.</p>
1687 <p>W32 uses three different calling conventions in its ABI. They are all
1689 <a href="http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vclang/html/_core_argument_passing_and_naming_conventions.asp"><span class="productname">Microsoft</span> documentation</a>.
1690 There is always necessary to have the proper function declaration
1691 (prototype) in the caller scope to prevent all sorts of unexpected
1694 <p>Unfortunately some non-matching combinations of calling conventions
1695 result in hard to debug bugs: the caller gets back an unexpected stack
1696 pointer from the callee and upon return it will restore registers from the
1697 wrong stack pointer place. Since the caller will finally reclaim its stack
1698 frame from its (uncorrupted) <span class="constant">EBP</span> stack frame
1699 pointer the caller will return to the caller of the caller correctly. Just
1700 the registers remain corrupted causing crashes of completely unrelated code
1701 executed far, far away...</p>
1703 <p><span class="constant">EDI</span>, <span class="constant">ESI</span> and
1704 <span class="constant">EBX</span> registers are always saved on the stack.
1705 They are stored on the stack in this particular order from bottom to top
1706 addresses (using the <span class="instruction">push EBX</span>,
1707 <span class="instruction">push ESI</span>,
1708 <span class="instruction">push EDI</span> sequence). Fortunately $gnulinux
1709 GCC has the same register saving behaviour. If some register corruption
1710 occurs the calling type presented between the caller and callee should be
1713 <a name="calltype_cdecl"><h3>W32 Calling Convention "cdecl"</h3></a>
1715 <p>The only calling convention in the UNIX world. The default one for all
1716 the compilers. All the arguments are passed on the stack, no arguments
1717 are cleaned by the callee. Possible inconsistencies in the number of
1718 function arguments with the function prototype used by the caller is
1719 harmless. Variable arguments lists can be passed by this convention.</p>
1721 @{[ doc_img 'fig/calltype_cdecl',
1722 'W32 Calling Convention <span class="constant">cdecl</span> Scheme' ]}
1724 <table border="1" align="center">
1725 <tr><td>Arguments freed by </td><td>caller</td></tr>
1726 <tr><td>Arguments on the stack </td><td>#0 ... #(n-1)</td></tr>
1727 <tr><td>Arguments in the registers </td><td>none</td></tr>
1728 <tr><td>GCC attribute </td><td><span class="command">__attribute__((__cdecl__))</span> (default)</td></tr>
1729 <caption>Calling Convention <span class="constant">cdecl</span> Characteristics</caption>
1732 <h3>W32 Calling Convention "stdcall"</h3>
1734 @{[ doc_img 'fig/calltype_stdcall',
1735 'W32 Calling Convention <span class="constant">stdcall</span> Scheme' ]}
1737 <p>Convention never used in the UNIX world. It needs to be specified for
1738 W32 compilers. All the arguments are passed on the stack, all the
1739 arguments are cleaned by the callee. Possible inconsistencies in the
1740 number of function arguments with the function prototype used by the
1741 caller will result in fatal crash. Variable arguments lists cannot be
1742 passed by this convention – use @{[ a_href '#calltype_cdecl','cdecl' ]}
1745 <table border="1" align="center">
1746 <tr><td>Arguments freed by </td><td>callee</td></tr>
1747 <tr><td>Arguments on the stack </td><td>#0 ... #(n-1)</td></tr>
1748 <tr><td>Arguments in the registers </td><td>none</td></tr>
1749 <tr><td>GCC attribute </td><td><span class="command">__attribute__((__stdcall__))</span></td></tr>
1750 <caption>Calling Convention <span class="constant">stdcall</span> Characteristics</caption>
1753 <h3>W32 Calling Convention "fastcall"</h3>
1755 <p>Convention never used in the UNIX world. It needs to be specified for
1756 W32 compilers. Convention used in the W32 world for its low calling
1757 overhead. All but the first two arguments are passed on the stack, such
1758 arguments are cleaned by the callee. First two arguments are passed in
1759 the registers <span class="constant">ECX</span> and
1760 <span class="constant">EDX</span> respectively. Possible inconsistencies
1761 in the number of function arguments with the function prototype used by
1762 the caller will result in fatal crash. Variable arguments lists cannot be
1763 passed by this convention – use @{[ a_href '#calltype_cdecl','cdecl' ]}
1766 <p>GCC (GNU C Compiler) native support for this calling convention
1767 is pretty fresh and it is currently present only in the recent CVS
1768 versions since 21st December of 2002 which should get released as GCC
1769 version 3.4. This project solved the unsupported calling convention by
1770 declaration of arguments passed in registers by
1771 <span class="command">__attribute__((__regparm__(3)))</span>.
1772 W32 passes the arguments in registers in the order
1773 <span class="constant">ECX</span>, <span class="constant">EDX</span> but
1774 GCC passes them in registers <span class="constant">EAX</span>,
1775 <span class="constant">EDX</span>, <span class="constant">ECX</span>.
1776 This incompatibility is compensated at C source level in the
1777 @{[ a_href '#functype','relaying code' ]} generated by
1778 <span class="fname">captivesym</span> relay generator.</p>
1780 @{[ doc_img 'fig/calltype_fastcall',
1781 'W32 Calling Convention <span class="constant">fastcall</span> Scheme' ]}
1783 <table border="1" align="center">
1784 <tr><td>Arguments freed by </td><td>callee</td></tr>
1785 <tr><td>Arguments on the stack </td><td>#2 ... #(n-1)</td></tr>
1786 <tr><td>Arguments in the registers </td><td><span class="constant">ECX</span>=#0,
1787 <span class="constant">EDX</span>=#1</td></tr>
1788 <tr><td>GCC ≥3.4 attribute </td><td><span class="command">__attribute__((__fastcall__))</span></td></tr>
1789 <tr><td>GCC <3.4 attr. emulation</td><td><span class="command">__attribute__((__stdcall__))</span></td></tr>
1790 <tr><td> </td><td><span class="command">__attribute__((__regparm__(3) /* EAX,EDX,ECX */))</span></td></tr>
1791 <caption>Calling Convention <span class="constant">fastcall</span> Characteristics</caption>
1794 <a name="synchronous"><h2>Multithreading and Multiple Processors</h2></a>
1796 <p>W32 platform stands on its thorough architecture parallelism. It
1797 must lock all its objects to maintain coherence in presence of
1798 multithreading and multiple processors. Since the author of this project
1799 considers any parallel execution a serious obstacle for debugging the whole
1800 project architecture was designed to prevent any undeterministic behaviour.
1801 Therefore this projects always emulates uniprocessor
1802 <span class="productname">Microsoft Windows NT</span> kernel
1803 (<span class="constant">KeNumberProcessors</span> symbol is always 1),
1804 everything runs in the single initial thread/process and all the filesystem
1805 operations are performed as synchronous
1806 ("synchronous" by flags
1807 <span class="constant">FILE_SYNCHRONOUS_IO_ALERT</span>,
1808 <span class="constant">FO_SYNCHRONOUS_IO</span>,
1809 <span class="constant">IRP_SYNCHRONOUS_API</span>,
1810 <span class="constant">IRP_SYNCHRONOUS_PAGING_IO</span>,
1811 forced <span class="constant">TRUE</span> result of
1812 <span class="function">IoIsOperationSynchronous()</span>
1814 For several cases needed only by <span class="fname">ntfs.sys</span> there
1815 had to be supported asynchronous access
1816 (<span class="constant">STATUS_PENDING</span> return code) – parallel
1817 execution is emulated by GLib
1818 <span class="function">g_idle_add_full()</span> with
1819 <span class="function">g_main_context_iteration()</span> called during
1820 <span class="function">KeWaitForSingleObject()</span>.</p>
1821 Since there is a possibility a real W32 parallel threading would
1822 be yet needed in the future all the code that would be hit by W32
1823 multithreading capability is marked by
1824 <span class="constant">TODO:thread</span> comment.</p>
1826 <p>Multiple processors (SMP) support will never need to be implemented
1827 since uniprocessor W32 kernels apparently run the filesystem driver modules
1828 fine. As this project implements only the uniprocessor W32 kernel all the
1829 processor locking functions and structures such as
1830 <span class="constant">KSPIN_LOCK</span> etc. can be safely implemented as
1833 <p>Asynchronous callbacks registered for
1834 <span class="constant">IO_WORKITEM</span>s are passed as GLib idle
1835 functions by <span class="function">g_idle_add_full()</span>. Although they
1836 will probably never be executed during non-interactive project's batch
1837 executions it is the responsibility of W32 driver implementation to
1838 complete all the pending tasks before its W32 shutdown. Such W32 shutdown
1839 is done during cleanup of the project's execution by
1840 <span class="function">captive_shutdown()</span>.</p>
1842 <a name="paranoia"><h2>Paranoia Checks</h2></a>
1844 <p>A general approach of software projects development is to implement
1845 many internal sanity checks during the development stage but to produce the
1846 most optimized final release product without those debugging checks.</p>
1848 <p>Facilities for these practices can be seen in the standard
1849 C include files for example as function
1850 <span class="function">assert()</span> which gets disabled by the
1851 <span class="constant">NDEBUG</span> symbol used during the final optimized
1852 executable compilation. This project uses Gnome GLib messaging subsystem
1853 offering sanity checks discarded by symbols
1854 <span class="constant">G_DISABLE_ASSERT</span> and
1855 <span class="constant">G_DISABLE_CHECKS</span>.
1856 <span class="productname">Microsoft</span> also produces two versions of
1857 its products – regular customers use the "free build" (also
1858 called "retail") while the programmers should develop their code
1859 on the "checked build" product releases.</p>
1861 <p>As this project will always run unknown binary code of proprietary W32
1862 filesystem drivers, the code can never be trusted. Such code even runs in
1863 the same unprotected address space as its controlling UNIX code. Since
1864 there is not enough documentation for the W32 components of the system and
1865 also such documentation is usually misleading it can never be considered as
1866 100% emulation. Even in the final releases all the sanity checks
1867 implemented in this project should remain active as all the project's code
1868 always interacts with unknown and untrusted W32 binaries.</p>
1870 <p><span class="productname">Microsoft Windows NT</span> code is written in
1871 a foolproof style as it accepts even invalid input values, and which
1872 it usually corrects. This makes long-term debugging a pain as it hides
1873 sources of problems. "Checked build" releases were probably
1874 designed to fix this flaw by strict consistency checks but it did not reach
1875 its goals as such checks are usually missing in the code.</p>
1877 <p>This project has strict consistency checks across all the code to make
1878 the debugging phase easy enough. Failed sanity check is not always
1879 a bug – sometimes it just means the real W32 binary code is more
1880 benevolent than it could be expected according to the documentation and
1881 such sanity check gets removed for the next version build. In other cases
1882 the failed sanity checks mean the execution path for some unexpected
1883 arguments combination was not yet implemented by this project. I may also
1884 mean a bug, of course...</p>
1886 <p>Last but not least – never miss a possible sanity check as its
1887 later removal is in an order of magnitude cheaper than an uncaught
1888 invalid assumption. Failed assertion is not always a bug although it
1889 has to be fixed, of course.</p>
1892 <h2>STATUS_LOG_FILE_FULL</h2>
1894 <p>After writing approx. 1MB of data on NTFS test partition NTFS driver
1895 returns for any further write requests
1896 <span class="constant">STATUS_LOG_FILE_FULL</status> error code.
1897 Apparently it is caused by the fact this project is
1898 @{[ a_href '#synchronous','single-threaded' ]} and it ignores the spawn
1899 of parallel journalling thread during <span class="fname">ntfs.sys</span>
1902 <p>Fortunately <span class="fname">ntfs.sys</span> will clear its
1903 journalling log file during filesystem unmount. This project will therefore
1904 remount the volume if <span class="constant">STATUS_LOG_FILE_FULL</status>
1905 is detected to workaround missing journalling thread.</p>
1907 <p>Similiar behaviour can be seen during write of compressed files —
1908 the file gets written uncompressed and its compression will proceed only
1909 during the final filesystem unmount.</p>
1911 <p>For these reasons it was mandatory to support
1912 @{[ a_href '#parent_connector','transparent volume remounting' ]}.</p>
1915 <a name="parent_connector"><h2><span class="constant">ParentConnector</span> volume remounter</h2></a>
1917 <p>The sandbox master component of this project has control of restarting
1918 its sandbox slaves containing the W32 filesystem. Target goal of
1919 <span class="constant">ParentConnector</span> component is to transparently
1920 provide persistent view of files and directories over the sandboxed slaves
1921 being restarted.</p>
1923 <p>In the case of read-only operations it would be simple as we could only
1924 save our state of currently opened filesystem objects with their read
1925 file/directory offset. Write operations can be handled as the read-only
1926 ones as long as all the operations are successful. In the case of W32
1927 filesystem crash we loose all the past write operations. If we would redo
1928 all the write operations we could very easily invoke the same crash.
1929 Therefore we write:</p>
1931 <blockquote class="command">
1932 <p>Filesystem crash broke dirty object: FILE/PATH/NAME</p>
1935 <p>message to syslog and refuse any further operations with this
1938 @{[ doc_img 'dia/parent-connector','Parent Connector' ]}
1940 <p><span class="constant">HANDLE</span> represents W32 object open in
1941 existing W32 filesystem.<span class="constant">HANDLE</span> is created
1942 on-demand according to the saved state of the object (such as its
1943 pathname). Even the whole <span class="constant">VFS</span> sandbox slave
1944 is spawn on-demand if some object operation requests it.</p>
1946 <p>W32 filesystem crash can obviously occur at any moment - it generates
1947 @{[ a_href 'http://developer.gnome.org/doc/API/2.0/gobject/','GObject' ]}
1948 @{[ a_href 'http://developer.gnome.org/doc/API/2.0/gobject/gobject-Signals.html','signal' ]}
1949 <span class="constant">abort</span>. Successful filesystem unmount
1950 (even as the part of remount operation) must be first preceded by
1951 <span class="constant">detach</span> signal to close all existing
1952 W32 <span class="constant">HANDLE</span>s. After their close the filesystem
1953 gets the unmount requests. Only in the case all the close operations
1954 succeeded including the final filesystem unmount the signal
1955 <span class="constant">cease</span> can be activated to notify all the
1956 dirty (written) objects they are now clean. During this
1957 <span class="constant">cease</span> signal the project will also
1958 @{[ a_href '#safe_flush','flush' ]} the sandbox commit buffer to its
1959 underlying media.</p>
1961 <p>Objects never written remain in <span class="constant">clean</span>
1962 state and they can be transparently reopened even if W32 filesystem crash
1966 <h1>TODO: Fsck of NTFS</h1>
1968 <p>Currently this project does not support checking of data structures
1969 of NTFS volume as being provided by <span class="command">chkdsk.exe</span>
1970 in W32 environment and <span class="command">fsck</span> in UNIX OS.</p>
1972 <p>W32 has its disk checking functionality split to
1973 <span class="fname">untfs.dll</span> W32 userland library.
1975 @{[ a_href 'http://www.sysinternals.com/ntw2k/source/fmifs.shtml',
1976 'Chkdskx and Formatx' ]}
1977 by @{[ a_href 'http://www.sysinternals.com/aboutus.shtml',
1978 'Mark Russinovich' ]}.
1980 <p>I assume its execution falls completely
1981 @{[ a_href '#existing_emulation','out of scope' ]}
1982 of this project as it is W32 userland.</p>
1984 <p>This possibility was not yet investigated in any way.</p>
1987 <h1>TODO: NTFS Support for
1988 <span class="productname">@{[ a_href 'http://surprise.sourceforge.net/','Partition Surprise' ]}</span></h1>
1990 <p>Although there currently exists
1991 <span class="productname">@{[ a_href 'http://mlf.linux.rulez.org/mlf/ezaz/ntfsresize.html','ntfsresize' ]}</span>
1992 I am not sure whether it is really reliable for all NTFS filesystems.
1993 <span class="productname">@{[ a_href 'http://surprise.sourceforge.net/','Partition Surprise' ]}</span>
1994 is the only partition manager capable of safely resize the disk
1995 by using just the original W32 filesystem driver by full rebuild of
1996 filesystem metadata.
1997 Almost no file data blocks would be moved even on these generic filesystems
1998 as W32 supports <span class="constant">FSCTL_MOVE_FILE</span> request
2000 @{[ a_href 'http://www.sysinternals.com/ntw2k/info/defrag.shtml',
2001 'Inside Windows NT Disk Defragmenting' ]}
2002 by @{[ a_href 'http://www.sysinternals.com/aboutus.shtml',
2003 'Mark Russinovich' ]}.
2006 <h1>Related Projects</h1>
2008 <p>The usual solution for file exchange between $freespeech operating systems
2009 and <span class="productname">Microsoft Windows NT</span> is to use
2010 <span class="productname">FAT32</span> (<span class="productname">vfat</span>
2011 called in $gnulinux) partition and swap the files over it. This method is not
2012 very comfortable as you never have access to all the files of the other
2013 operating system.</p>
2015 <a name="LinuxNTFScompet"><h2>$LinuxNTFS</h2></a>
2017 <p>Although this project takes a completely different approach and has
2018 a different architecture, the final goal is the same as for this
2019 project – reliable read-write <span class="productname">NTFS</span>
2020 filesystem support. $LinuxNTFS goes the way of reverse engineering
2021 filesystem data structures (and possibly
2022 <span class="fname">ntfs.sys</span> itself). Unfortunately after many years
2023 of its development it did not yet reach the state of reliable read-write
2024 access although its read-only part is considered trustworthy.</p>
2026 <p>Using $LinuxNTFS for read-only access to existing partition with
2027 <span class="productname">Microsoft Windows NT</span> installation is
2028 planned to be able to acquire existing <span class="fname">ntfs.sys</span>,
2029 <span class="fname">ntoskrnl.exe</span> and possibly
2030 <span class="fname">ksecdd.sys</span> (imported by
2031 <span class="fname">ntfs.sys</span>) files from the user's
2032 <span class="productname">NTFS</span> partition.</p>
2034 <h2><span class="productname">@{[ a_href 'http://www.cgsecurity.org/ntfs.html','NTPwd NTFS Driver' ]}</span></h2>
2036 <p>DOS based @{[ a_href 'http://www.gnu.org/licenses/gpl.html','GPL-2.0' ]}
2037 read-write NTFS driver. Filesystem structures are reverse engineered in the
2038 way of @{[ a_href '#LinuxNTFScompet','Linux-NTFS Project' ]}. As it is not very
2039 actively maintained it reaches a lower level of
2040 <span class="productname">NTFS</span> compatibility.</p>
2042 <h2>@{[ a_href 'http://www.vmware.com/download/workstation.html','VMware Workstation' ]}</h2>
2044 <p>The only real competition: Closed-source read/write @{[ '$299' ]} equivalent.</p>
2046 <p>Original Microsoft Windows operating system can be run inside a virtual
2047 machine running under GNU/Linux and share the read-write NTFS disk by using
2048 a network file sharing through a VMware virtual network card.</p>
2050 <p>You need @{[ '$299' ]} for this product and you need to
2051 give up your system security by running un@{[ a_href '#sandbox','sandbox' ]}ed
2052 closed-source program in your GNU/Linux.</p>
2054 <h2>@{[ a_href 'http://www.winehq.com/','Wine Project' ]}</h2>
2056 <p>No code could be shared – Wine emulates only Microsoft Windows userland.
2057 Filesystem drivers completely belong to Microsoft Windows kernelland.</p>
2059 <h2>@{[ a_href 'http://www.sysinternals.com/ntw2k/freeware/ntfswin98.shtml','NTFS for Windows 98' ]}</h2>
2061 <p>Closed-source read-only-crippled @{[ '$0' ]} equivalent for Microsoft Windows.</p>
2063 <p>There is a @{[ a_href 'http://www.sysinternals.com/images/screenshots/ntfs98ap.gif',
2064 'diagram' ]} showing exactly the principle of Captive NTFS project.
2065 There is apparently disabled read/write functionality in <i>NTFS for
2066 Windows 98</i> as the same company also sells the following product sharing
2067 the same codebase:</p>
2069 <h2>@{[ a_href 'http://www.winternals.com/products/repairandrecovery/ntfsdospro.asp','NTFSDOS Professional' ]}</h2>
2071 <p>Closed-source read/write @{[ '$299' ]} equivalent for MS-DOS.</p>
2073 <p>This product is the most close equivalent to Captive NTFS but it is
2074 a commercial product, closed-source and it has filesystem interface only
2078 <h1>Re: @{[ a_href 'http://linux-ntfs.sourceforge.net/info/ntfs.html#7.7',
2079 "7.7 Can't we write a wrapper for Windows' driver?" ]}</h1>
2081 <p class="re">> It sounds like a great idea, to start with, but there are numerous
2084 <p><span class="re">> The largest technical problem is joining the Windows
2085 system DLL to the Linux VFS. It could be done, but it wouldn't be pretty.</span><br />
2088 <p><span class="re">> It would have to run as part of the kernel which would mean
2089 that if it went wrong it could crash the machine. With no source, we might not
2090 be able to work around the problem.</span><br />
2091 @{[ a_href '#sandbox','Nope' ]},
2092 @{[ a_href 'http://lufs.sourceforge.net/lufs/','Linux Userland File System (LUFS)' ]}
2093 moves the filesystem implementation to UNIX userland where the Microsoft
2094 Windows filesystem is completely unarmed by Captive jail of chroot(2),
2095 setuid(2) and setrlimit(2). There only remains one narrow connection to the rest of
2096 system (by CORBA/ORBit). The filesystem's life environment gets kill(2)ed when
2097 UNIX is no longer satisfied with it. Safety similiar to
2098 @{[ a_href 'http://www.vmware.com/solutions/security.html','VMware sandbox' ]}.</p>
2100 <p><span class="re">> The next major problem is compati<!--orig. text typo-->bility.
2101 Which version of the Windows system file would we use? Picking one would limit
2102 its use, making the wrapper versatile for all of them would be a programming
2103 nightmare.</span><br />
2104 Microsoft Windows NTFS filesystem driver is capable of accessing older formats
2105 of the filesystem. This project currently runs Microsoft Windows XP version,
2106 porting to Microsoft Windows 2003 Server expected. (Microsoft Windows upgrades
2107 NTFS disk filesystem to its own version during complete CD-ROM Microsoft
2108 Windows system installation – such operation is not threat this project use.)</p>
2110 <p><span class="re">> And it gets worse. The legal implications of
2111 distributing Windows systems files would cause problems.</span><br />
2112 User must be careful to obey all licensing restrictions according to his
2113 local country laws.<br />
2114 <span class="re">> Also the proprietary nature of the driver would mean that
2115 the other kernel coders would not investigate any problems if someone had used
2116 the NTFS wrapper.</span><br />
2117 It does not apply to this project due to the implemented
2118 @{[ a_href '#sandbox','filesystem separation' ]}.</p>