4 # Captive project doc Components page Perl template.
5 # Copyright (C) 2003 Jan Kratochvil <project-www.jankratochvil.net@jankratochvil.net>
7 # This program is free software; you can redistribute it and/or modify
8 # it under the terms of the GNU General Public License as published by
9 # the Free Software Foundation; exactly version 2 of June 1991 is required
11 # This program is distributed in the hope that it will be useful,
12 # but WITHOUT ANY WARRANTY; without even the implied warranty of
13 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
14 # GNU General Public License for more details.
16 # You should have received a copy of the GNU General Public License
17 # along with this program; if not, write to the Free Software
18 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
21 package project::captive::doc::Components;
22 require 5.6.0; # at least 'use warnings;' but we need some 5.6.0+ modules anyway
23 our $VERSION=do { my @r=(q$Revision$=~/\d+/g); sprintf "%d.".("%03d"x$#r),@r; };
28 BEGIN{ open F,"Makefile"; our $top_dir=pop @{[split /\s/,(grep /^top_srcdir/,<F>)[0]]}; eval "use lib '$top_dir'"; close F; }
31 use project::captive::doc::Macros;
35 "__PACKAGE__"=>__PACKAGE__,
36 "title"=>'Captive NTFS Developer Documentation: Components',
37 "head_css"=>$doc_Macros_head_css,
45 <h1>Project Components</h1>
47 @{[ doc_img 'dia/arch-all','Project Components Architecture' ]}
49 <p>Most of the work of this project is located in the single box called
50 "<span class="constant">libcaptive</span>" located in the center
51 of the scheme. This component implements the core W32 kernel API by
52 various methods described in this document.
53 The "<span class="constant">libcaptive</span>" box cannot be
54 further dissected as it is just an implementation of a set of
55 @{[ captive_srcfile 'src/libcaptive/ke/exports.captivesym','API functions' ]}.
56 It could be separated to several subsystems such as the
57 @{[ a_href '#cache_manager','Cache Manager' ]},
58 Memory Manager, Object Manager, Runtime Library, I/O Manager
59 etc. but they have no interesting referencing structure.</p>
61 <p>As this project is in fact just a filesystem implementation every
62 story must begin at the device file and end at the filesystem operations
63 interface. The unified suppported interfaces are
64 <span class="productname">@{[ a_href 'http://developer.gnome.org/doc/API/2.0/glib/','GLib' ]}</span>
65 (the most low level portability, data-types and utility library for Gnome)
66 <span class="type">GIOChannel</span> (for the device access) and the custom
67 <span class="constant">libcaptive</span> filesystem API. Each of these ends
68 can be connected either to some direct interface (such as the
69 <span class="constant">captive-cmdline</span> client),
70 @{[ a_href 'http://lufs.sourceforge.net/lufs/','Linux Userland File System (LUFS)' ]}
71 or as a general $GnomeVFS filter.
72 @{[ a_href 'http://lufs.sourceforge.net/lufs/','LUFS' ]} will be used in
73 most cases as it offers standard filesystem interface by Linux kernel.
75 You can also use $GnomeVFS as it offers nice filter interface on
76 the UNIX user-privileges level for transparent operation with archives and
77 network protocols. This filter interface was used by this project to turn
78 the device reference such as <span class="fname">/dev/hda3</span> or <span
79 class="fname">/dev/discs/disc0/part3</span> to the fully accessible
80 filesystem (pretending being an "archive" in the device
81 reference). This device access can be specified by $GnomeVFS URLs such as:
83 class="fname">file:///dev/hda3#captive-fastfat:/autoexec.bat</span></p>
85 <span class="constant">captive-bug-replay</span> serves just for debugging
86 purposes — you can 'replay' existing
87 <span class="fname">file.captivebug.xml.gz</span> automatically being
88 generated during W32 filesystem failure. This bugreport file will contain
89 all the touched data blocks of the device used in the moment of the
90 failure. <span class="constant">captive-bug-replay</span> will therefore
91 emulate internal virtual writable device out of these bugreported data.
93 <p>If the passed device reference is requested by the user to be accessed
94 either in <span class="dashdash">--ro</span> (read-only) mode or in the
95 <span class="dashdash">--rw</span> (full read-write) mode there are no
96 further device layers needed. Just in the case of <span
97 class="dashdash">--blind</span> mode another layer is involved to emulate
98 read-write device on top of the real read-only device by the method of
99 non-persistent memory buffering of all the possible write requests.</p>
101 <span class="constant">sandbox commit buffer</span> is involved only in the
102 case @{[ a_href 'Details.html.pl#sandbox','sandboxing feature' ]} is active. It will
103 buffer any writes to the device during the sandbox run to prevent
104 filesystem damage if the driver would fail in the meantime. If the
105 filesystem gets finally successfully unmounted this sandbox buffer can be
106 <a name="safe_flush">safely flushed</a>
107 to its underlying physical media. The buffer will be dropped
108 in the case of filesystem failure, of course. The filesystem should be
109 unmounted from time to time — it can be transparently unmounted and mounted
110 by <span class="command">commit</span> of
111 <span class="constant">captive-cmdline</span> custom client. Currently you
112 cannot force remounting when using
113 @{[ a_href 'http://lufs.sourceforge.net/lufs/','LUFS' ]} interface client
114 but it will be remounted after approx each 1MB data written automatically
115 due to @{[ a_href '#log_file_full','NTFS log file full' ]}.
117 Now we need to transparently
118 @{[ captive_srcfile 'src/libcaptive/sandbox/sandbox.idl','connect' ]}
119 the device interface of <span class="type">GIOChannel</span> type through
120 @{[ a_href 'Details.html.pl#sandbox','CORBA/ORBit' ]} to the sandboxed slave.
122 <p>Such device is still only a UNIX style GLib <span
123 class="type">GIOChannel</span> type at this point. As we need to supply it
124 to the W32 filesystem driver we must convert it to the W32 I/O Device
125 with its capability of handling <span class="type">IRP</span>
126 (<span class="constant">I/O Request Packet</span>; structure holding the
127 request and result data for any W32 filesystem or W32 block device
129 requests from its upper W32 filesystem driver. Such W32 I/O Device can
130 represent either <span class="type">CD-ROM</span> or
131 <span class="type">disk</span> device type as different W32 filesystem
132 drivers require different media types — currently only
133 <span class="fname">cdfs.sys</span> requires
134 <span class="type">CD-ROM</span> type.</p>
136 <p>W32 media I/O Device is accessed from the W32 filesystem driver.
137 The filesystem driver itself always creates volume object by
138 <span class="function">IoCreateStreamFileObject()</span> representing the
139 underlying W32 media I/O Device as the object handled by the
140 filesystem driver itself. All the client application filesystem requests
141 must be first resolved at the filesystem structures level, passed to the
142 volume stream object of the same filesystem and then finally passed to the
143 W32 media I/O Device (already implemented by this project as an
144 interface to <span class="type">GIOChannel</span> noted above).</p>
146 <p>The filesystem driver is called by the core W32 kernel implementation of
147 <span class="constant">libcaptive</span> in
148 @{[ a_href 'Details.html.pl#synchronous','synchronous way' ]} in single-shot manner instead of
149 the several reentrancies while waiting for the disk I/O completions as can
150 be seen in the original
151 <span class="productname">Microsoft Windows NT</span>.
152 This single-shot synchronous behaviour is possible since all the needed
153 resources (disk blocks etc.) can be always presented as instantly ready as
154 their acquirement is solved by @{[ a_href 'hostosnote','Host-OS' ]} outside of
155 the W32 emulated @{[ a_href 'guestosnote','Guest-OS' ]} environment.
156 For several cases needed only by <span class="fname">ntfs.sys</span>
157 there had to be supported asynchronous access — parallel execution
158 is emulated by GLib <span class="function">g_idle_add_full()</span>
159 with <span class="function">g_main_context_iteration()</span> called during
160 <span class="function">KeWaitForSingleObject()</span>.</p>
162 <p><span class="constant">libcaptive</span> offers the W32 kernel
163 filesystem API to the upper layers. This is still not the API the common
164 W32 applications are used to as they use W32 libraries which in turn pass
165 the call to W32 kernel. For example
166 <span class="function">CreateFileA()</span> is being implemented by several
167 libraries such as <span class="fname">user32.dll</span> as a relay
168 interface for the kernel function
169 <span class="function">IoCreateFile()</span> implemented by this
170 project's <span class="constant">libcaptive</span> W32 kernel
171 emulation component.</p>
173 <p>As it would be very inconvenient to use the legacy, bloated and UNIX
174 style unfriendly W32 kernel filesystem API this project offers its own
175 @{[ a_href '#client_interface','custom filesystem API interface' ]} inspired by
176 the $GnomeVFS client interface adapted to the specifics of W32 kernel API.
177 This interface is supposed to be easily utilized by
178 <a href="#client_interface_customapp">a custom application accessing
179 the W32 filesystem driver</a>.</p>
181 <p>@{[ a_href 'Details.html.pl#sandbox','CORBA/ORBit' ]} hits us again – we need to
182 @{[ captive_srcfile 'src/libcaptive/sandbox/sandbox.idl','translate' ]}
183 the @{[ a_href '#client_interface','custom filesystem API interface' ]}
184 out of the sandboxed slave to the UNIX space.</p>
186 <p><span class="constant">captive sandbox master</span> provides the
187 functionality of covering any possible sandboxed slave restarts and its
188 communication. It is also capable of
189 <a name="demultiplexing_master">demultiplexing single API operations</a>
190 to multiple its connected sandbox slaves in transparent way
191 as each of them handles
192 @{[ a_href 'Details.html.pl#mounted_one','just one filesystem device' ]}.</p>
194 <p>The rest of the story is not much special for this project since this is
195 a common UNIX problem how to offer user space implemented UNIX filesystem
196 as a generic system filesystem (as those are usually implemented only as
197 the components od UNIX kernel).</p>
199 <p>The filesystem service can be offered in several ways:</p>
202 <dt>Custom client</dt>
204 <p>One possibility would be to write
205 <a name="client_interface_customapp">a custom client application</a>
206 for this project such as file manager or a shell. Although it
207 would implement the most appropriate user interface to the set of
208 functions offered by this project (and W32 filesystem API) it has the
209 disadvantage of special client software. Appropriate client is provided
211 <span class="fname">src/client/cmdline/cmdline-captive</span></p>
214 <dt>@{[ a_href 'http://lufs.sourceforge.net/lufs/','Linux Userland File System (LUFS)' ]}</dt>
216 <p>The most usable interface is the
217 @{[ a_href 'http://lufs.sourceforge.net/lufs/','LUFS' ]} client
218 by <span class="constant">liblufs-captivefs</span>.
219 As @{[ a_href 'http://lufs.sourceforge.net/lufs/','LUFS' ]}
220 already assigns separate process for each filesystem mount the
221 @{[ a_href '#demultiplexing_master','demultiplexing feature' ]}
222 is not utilized in this case.</p>
224 <p>@{[ a_href 'http://lufs.sourceforge.net/lufs/','LUFS' ]}
225 needs multiple operating threads (each UNIX kernel operation needs
226 one free lufsd slot/thread to not to fail immediately).
227 As <span class="constant">libcaptive</span> is
228 @{[ a_href 'Details.html.pl#synchronous','single-threaded' ]} all the operations
229 get always synchronized by
230 <span class="constant">liblufs-captivefs</span>
231 before their pass over to <span class="constant">libcaptive</span>.</p>
234 <dt>@{[ a_href '#offered_gnomevfs','Gnome-VFS' ]}</dt>
236 <p>This client allowing its filesystem access even without any
237 involvement of UNIX kernel from any $GnomeVFS aware client application
238 (such as <span class="fname">gnome-vfs/tests/test-shell</span>).
239 This @{[ a_href '#offered_gnomevfs','Gnome-VFS interface' ]} connects the
240 data flow of this project in two points — both as the lowest layer
241 device image source and also as the upper layer for the filesystem
242 operation requests.</p>
246 <p>Unimplemented and deprecated methods for providing filesystem
250 <dt>W32 filesystem in UNIX OS kernel</dt>
252 <p>The real UNIX OS filesystem implementation must be completely
253 implemented inside the hosting OS kernel. This requires special coding
254 methods with limited availability of coding features and libraries.
255 Also it would give the full system control to the untrusted W32
256 filesystem driver code with possibly fatal consequences of yet
257 unhandled W32 emulation code paths. It would benefit from the best
258 execution performance but this solution was never considered a real
262 <dt>Custom NFS server</dt>
264 <p>The common approach
265 <a name="offered_NFS">of filesystem implementations</a>
266 outside UNIX OS kernel were custom NFS servers usually running on the
267 same machine as the NFS-connected client as such NFS server is usually
268 an ordinary UNIX user space process. It would be possible to implement
269 this project as a custom NFS server but the NFS protocol itself
270 has a lot of fundamental flaws and complicated code for backward