Custom stream with 64bit streampos? - c++

This is a discussion on Custom stream with 64bit streampos? - c++ ; I'm trying to do work with large files (>2GB), and I'd like to use stream-like classes in my code as much as possible. Unfortunately, I'm developing with a 32 bit version of VC2005, which defines the streampos, streamsize, and streamoff ...

+ Reply to Thread
Results 1 to 6 of 6

Custom stream with 64bit streampos?

  1. Default Custom stream with 64bit streampos?

    I'm trying to do work with large files (>2GB), and I'd like to use
    stream-like classes in my code as much as possible. Unfortunately,
    I'm developing with a 32 bit version of VC2005, which defines the
    streampos, streamsize, and streamoff types using 32 bit integers.

    I don't have any problem with defining a new class that uses system
    calls to do the disk reads and writes, but I'd like it to share as
    many qualities with the basic_ios derivatives as possible.
    Particularly, I'd like to be able to change all references to
    std::istream to MyIstream64 without needing to completely refactor all
    of my existing code, because MyIstream64 will have an identical
    interface.

    I don't need to be able to cast the 64 bit stream to a basic_stream;
    I'm alright with the fact that my stream will have different internal
    than those written by MS. I'm really only interested in making a
    class where I have complete control of the filebuf implementation.
    That way I can write code that implements operator>> by reusing the
    ios classes for formatted character streams. My ultimate goal is to
    write code that I can quickly update when I move to a compiler that
    supports 64 bit stream offsets, without having to write all of the
    formatting code myself. To the best of my understanding, formatting a
    stream of characters into an integer shouldn't really care about the
    position of the read pointer in the stream anyhow.

    Is something like this possible? I've spent some time trying to
    disect all of the relationships between streams, char traits, stream
    buffers, and the locale system. I'm not sure how I'd work out methods
    like rdbuf, even if I only support redirection between my 64 bit
    streams. Still, it's a lot to wrap my mind around, so I figured I'd
    ask the experts and pray.

    Thanks,
    Chase


    --
    [ See http://www.gotw.ca/resources/clcm.htm for info about ]
    [ comp.lang.c++.moderated. First time posters: Do this! ]


  2. Default Re: Custom stream with 64bit streampos?

    chase.bradford@saic.com a écrit :
    > I'm trying to do work with large files (>2GB), and I'd like to use
    > stream-like classes in my code as much as possible. Unfortunately,
    > I'm developing with a 32 bit version of VC2005, which defines the
    > streampos, streamsize, and streamoff types using 32 bit integers.
    >
    > I don't have any problem with defining a new class that uses system
    > calls to do the disk reads and writes, but I'd like it to share as
    > many qualities with the basic_ios derivatives as possible.


    The usual way to do that is to define a new streambuf that fit your
    needs and wrap it in i/o streams or use it directly with istream and
    ostream (that take a streambuf in parameters).

    Additionnaly, you may have to use specifie char_traits for 64 bits
    operations. I am not sure about that (I will use std::char_traits<> in
    the examples).

    I assume you will use specific 64bits file access function.

    1. Define the custom streambuf
    template <typename CharT, typename Traits = std::char_traits<CharT> >
    class basic_filebuf64
    : public std::basic_streambuf<CharT, Traits>
    {
    public:
    // Define traits - make sure they are 64 bit
    typedef CharT char_type;
    typedef Traits traits_type;
    typedef typename traits_type::int_type int_type;
    typedef typename traits_type:ff_type off_type;
    typedef typename traits_type:os_type pos_type;

    //implement streambuf functions
    // add file specific operation open, close ...
    // see std::filebuf for an example
    //...
    };

    2. Then the input stream
    template <typename CharT, typename Traits = std::char_traits<CharT> >
    class basic_ifstream64
    : public std::basic_istream<CharT, Traits>
    {
    typedef std::basic_istream<CharT, Traits> istream_type;

    //constructor of istream_type takes pointer on filebuf64

    //delagate open/close/... operations to composed basic_bilebuf64
    };

    3. And the output stream
    template <typename CharT, typename Traits = std::char_traits<CharT> >
    class basic_ofstream64
    : public std::basic_ostream<CharT, Traits>
    {
    typedef std::basic_ostream<CharT, Traits> ostream_type;
    //constructor of ostream_type takes pointer on filebuf64

    //delagate open/close/... operations to composed basic_bilebuf64
    };

    4. Perhaps, the bidirectionnal stream
    template <typename CharT, typename Traits = std::char_traits<CharT> >
    class basic_fstream64
    : public std::basic_iostream<CharT, Traits>
    {
    typedef std::basic_iostream<CharT, Traits> iostream_type;
    //constructor of iostream_type takes pointer on filebuf64

    //delagate open/close/... operations to composed basic_bilebuf64
    };

    5. Finally, specialize the templates
    typedef basic_filebuf64<char> filebuf64;
    typedef basic_ifstream64<char> ifstream64;
    typedef basic_ofstream64<char> ofstream64;



    > Particularly, I'd like to be able to change all references to
    > std::istream to MyIstream64 without needing to completely refactor all
    > of my existing code, because MyIstream64 will have an identical
    > interface.


    A typedef should be enough
    #if // ON 32 bits architecture
    typdef myns::ifstream64 myfstream;
    #else
    typdef std::ifstream myfstream;
    #endif

    If you really need to change a whole context, you can try changing
    namespace alias conditionnaly. I have rarely found this practical.

    > I don't need to be able to cast the 64 bit stream to a basic_stream;
    > I'm alright with the fact that my stream will have different internal
    > than those written by MS. I'm really only interested in making a
    > class where I have complete control of the filebuf implementation.
    > That way I can write code that implements operator>> by reusing the
    > ios classes for formatted character streams. My ultimate goal is to
    > write code that I can quickly update when I move to a compiler that
    > supports 64 bit stream offsets, without having to write all of the
    > formatting code myself. To the best of my understanding, formatting a
    > stream of characters into an integer shouldn't really care about the
    > position of the read pointer in the stream anyhow.


    No. This is the task of the underlying streambuf.

    > Is something like this possible? I've spent some time trying to
    > disect all of the relationships between streams, char traits, stream
    > buffers, and the locale system. I'm not sure how I'd work out methods
    > like rdbuf, even if I only support redirection between my 64 bit
    > streams. Still, it's a lot to wrap my mind around, so I figured I'd
    > ask the experts and pray.


    Try google for "streambuf custom", there should be some example.

    Michael



    --
    [ See http://www.gotw.ca/resources/clcm.htm for info about ]
    [ comp.lang.c++.moderated. First time posters: Do this! ]


  3. Default Re: Custom stream with 64bit streampos?

    chase.bradford@saic.com wrote:
    > I'm trying to do work with large files (>2GB), and I'd like to use
    > stream-like classes in my code as much as possible. Unfortunately,
    > I'm developing with a 32 bit version of VC2005, which defines the
    > streampos, streamsize, and streamoff types using 32 bit integers.


    Umm, this surprises me. Even with a 32bit target, filesizes > 2^32 have been
    common for years and a VS2005 is a rather new product, so the first thing
    I'd suggest is to check if it really only supports 32 file positions or if
    it is maybe just a #define away. You might be able to get some info from
    either Dinkumware directly (they supply the implementation for VS) or post
    in microsoft.public.stl or so (forgot its exact name), which is also read
    by several Dinkumware people.

    Other than that, you could simply replace the whole C++ standardlibrary. I'm
    pretty sure that e.g. STLport does support 64bit file sizes under win32. If
    you don't want to replace the whole of it, even that is possible, as you
    can configure STLport to remain in its own namespace, so you would simply
    use stlport::istream instead of std::istream.

    > I don't have any problem with defining a new class that uses system
    > calls to do the disk reads and writes, but I'd like it to share as
    > many qualities with the basic_ios derivatives as possible.
    > Particularly, I'd like to be able to change all references to
    > std::istream to MyIstream64 without needing to completely refactor all
    > of my existing code, because MyIstream64 will have an identical
    > interface.

    [...]
    > Is something like this possible?


    Yes, but it probably boils down to a complete reimplementation.

    > I've spent some time trying to
    > disect all of the relationships between streams, char traits, stream
    > buffers, and the locale system. I'm not sure how I'd work out methods
    > like rdbuf, even if I only support redirection between my 64 bit
    > streams. Still, it's a lot to wrap my mind around, so I figured I'd
    > ask the experts and pray.


    Depending on what and how much of the stream interface you need, you might
    be able to write a simple class that behaves like a stream and does just
    what you need and only uses what you use. If you are using all the complex
    features like redirection, locales/facets, callbacks you are better off
    using a 64bit-capable implementation.

    Uli


    --
    [ See http://www.gotw.ca/resources/clcm.htm for info about ]
    [ comp.lang.c++.moderated. First time posters: Do this! ]


  4. Default Re: Custom stream with 64bit streampos?

    On Jul 17, 11:43 am, Ulrich Eckhardt <dooms...@knuut.de> wrote:
    > chase.bradf...@saic.com wrote:
    > > I'm trying to do work with large files (>2GB), and I'd like to use
    > > stream-like classes in my code as much as possible. Unfortunately,
    > > I'm developing with a 32 bit version of VC2005, which defines the
    > > streampos, streamsize, and streamoff types using 32 bit integers.

    >
    > Umm, this surprises me. Even with a 32bit target, filesizes > 2^32 have

    been
    > common for years and a VS2005 is a rather new product, so the first thing
    > I'd suggest is to check if it really only supports 32 file positions or if
    > it is maybe just a #define away. You might be able to get some info from
    > either Dinkumware directly (they supply the implementation for VS) or post
    > in microsoft.public.stl or so (forgot its exact name), which is also read
    > by several Dinkumware people.
    >


    I thought that VS2005's implementation could support files like those
    as well, but the pos_type in char_traits<char>'s is a synonym for
    std::streampos. When _WIN64 is defined, the type is a 64 bit int,
    otherwise it's only 32.

    I think I'll try Michael's solution, because that was already the
    direction I was heading. If nothing else, this will be a good time to
    get a better understanding of the streams' inner-workings.

    Thanks
    Chase


    --
    [ See http://www.gotw.ca/resources/clcm.htm for info about ]
    [ comp.lang.c++.moderated. First time posters: Do this! ]


  5. Default Re: Custom stream with 64bit streampos?

    chase.bradford@saic.com a écrit :
    > On Jul 17, 11:43 am, Ulrich Eckhardt <dooms...@knuut.de> wrote:
    >> chase.bradf...@saic.com wrote:
    >>> I'm trying to do work with large files (>2GB), and I'd like to use
    >>> stream-like classes in my code as much as possible. Unfortunately,
    >>> I'm developing with a 32 bit version of VC2005, which defines the
    >>> streampos, streamsize, and streamoff types using 32 bit integers.

    >> Umm, this surprises me. Even with a 32bit target, filesizes > 2^32 have

    > been
    >> common for years and a VS2005 is a rather new product, so the first thing
    >> I'd suggest is to check if it really only supports 32 file positions or

    if
    >> it is maybe just a #define away. You might be able to get some info from
    >> either Dinkumware directly (they supply the implementation for VS) or

    post
    >> in microsoft.public.stl or so (forgot its exact name), which is also read
    >> by several Dinkumware people.
    >>

    >
    > I thought that VS2005's implementation could support files like those
    > as well, but the pos_type in char_traits<char>'s is a synonym for
    > std::streampos. When _WIN64 is defined, the type is a 64 bit int,
    > otherwise it's only 32.
    >
    > I think I'll try Michael's solution, because that was already the
    > direction I was heading. If nothing else, this will be a good time to
    > get a better understanding of the streams' inner-workings.


    However, you will have to define 64 bits char_traits.
    Personnaly, I would first define the 64 bits char_traits and try them in
    the usual basic_fstream<> template.

    typedef basic_fstream <char,chr_traits64<char> > fstream64;
    typedef basic_ifstream<char,chr_traits64<char> > ifstream64;
    typedef basic_ofstream<char,chr_traits64<char> > ofstream64;

    If that doesn't work because the underlying implementation of
    basic_fstream do not support 64 bits access to files(which is likely), I
    would then redefine a 64 bit fstream as I mentionned earlier.

    Under Linux there is a #define to set in order (#define
    _LARGEFILE64_SOURCE 1) to have 64 bit files. As Chase BradFord said, I
    expect the same exists under Windows and that would be far quicker.

    Michael


    --
    [ See http://www.gotw.ca/resources/clcm.htm for info about ]
    [ comp.lang.c++.moderated. First time posters: Do this! ]


  6. Default Re: Custom stream with 64bit streampos?

    chase.bradford@saic.com wrote:
    :: On Jul 17, 11:43 am, Ulrich Eckhardt <dooms...@knuut.de> wrote:
    ::: chase.bradf...@saic.com wrote:
    :::: I'm trying to do work with large files (>2GB), and I'd like to
    :::: use stream-like classes in my code as much as possible.
    :::: Unfortunately, I'm developing with a 32 bit version of VC2005,
    :::: which defines the streampos, streamsize, and streamoff types
    :::: using 32 bit integers.
    :::
    ::: Umm, this surprises me. Even with a 32bit target, filesizes >
    ::: 2^32 have been common for years and a VS2005 is a rather new
    ::: product, so the first thing I'd suggest is to check if it really
    ::: only supports 32 file positions or if it is maybe just a #define
    ::: away. You might be able to get some info from either Dinkumware
    ::: directly (they supply the implementation for VS) or post in
    ::: microsoft.public.stl or so (forgot its exact name), which is also
    ::: read by several Dinkumware people.
    :::
    ::
    :: I thought that VS2005's implementation could support files like
    :: those as well, but the pos_type in char_traits<char>'s is a
    :: synonym for std::streampos. When _WIN64 is defined, the type is a
    :: 64 bit int, otherwise it's only 32.

    No, streampos is a structure (class fpos<>) that holds the actual file
    position. It is streamoff that is limited to the largest available
    integer type - long.


    The library DOES support larger files as such. It all depends on what
    you intend to do.

    If you are just streaming data, it works ok. If you want relative
    positioning, this is limited to +/- 2GB offsets. If you want to seek
    to a previously saved position, that is not limited.


    Bo Persson



    --
    [ See http://www.gotw.ca/resources/clcm.htm for info about ]
    [ comp.lang.c++.moderated. First time posters: Do this! ]


+ Reply to Thread

Similar Threads

  1. streambuf :: getting the data back from a custom stream
    By Application Development in forum c++
    Replies: 4
    Last Post: 11-11-2007, 05:19 AM
  2. Replies: 0
    Last Post: 05-23-2007, 10:45 AM
  3. Replies: 0
    Last Post: 03-22-2007, 06:36 AM
  4. Can to convert string in binary stream to text stream???
    By Application Development in forum basic.visual
    Replies: 9
    Last Post: 01-08-2007, 12:14 PM
  5. 64bit/64bit fixed point division?
    By Application Development in forum ASM x86 ASM 370
    Replies: 10
    Last Post: 08-26-2006, 12:41 PM