Memory use by MSXML

Hi, I'm using MSXML4 and I'm having an issue with memory allocation which
appears to be a memory leak, but isn't exactly. Let me explain.

I have this one XML file (the "IXM" file) that contains the names of 148
other XML files (the "external" files). What is supposed to happen is that
the app reads the IXM file, then reads each external file named in the IXM
file, then copies the external file's data to the database. The problem is
that after about 36 external files, it runs out of memory. Each external file
is generally between 104 MB and 106 MB, but some are larger. The largest is
136 MB. The external files are in ASCII.

I run performance monitor and monitor virtual bytes. It seems that the
virtual bytes are increasing all the time, and by the time of the 36th
external file, virtual bytes tries to exceed 2 GB, and it bombs.

Based on experiments I've done, I conclude the following:

1. When MSXML is allocating memory, it is not doing so on the process heap,
but on its own heap, so memory looks like this:

[PROCESS-HEAP][HEAP-1][HEAP-2][MSXML-HEAP-1][MSXML-HEAP-2]

2. The SAX parser in MSXML reads the entire external file into a contiguous
region of memory before it will call the content handler.

3. If MSXML cannot load the entire external file into a contiguous region of
memory in its own heaps, it allocates another heap:

[PROCESS-HEAP][HEAP-1][HEAP-2][MSXML-HEAP-1][MSXML-HEAP-2][MSXML-HEAP-3]

[PROCESS-HEAP][HEAP-1][HEAP-2][MSXML-HEAP-1][MSXML-HEAP-2][MSXML-HEAP-3][MSXML-HEAP-4]

[PROCESS-HEAP][HEAP-1][HEAP-2][MSXML-HEAP-1][MSXML-HEAP-2][MSXML-HEAP-3][MSXML-HEAP-4][MSXML-HEAP-5]

It does not seem to be freeing the extra heaps that it is creating. So
eventually, it will try to allocate a memory block that's too big for the
existing heaps and too big to allocate a new heap.

Assuming that's the case:

1. How do I free up all the heaps that MSXML allocates, preferably between
external files?

2. How do I get MSXML to use the app's heap instead of creating its own?

3. Is there any other strategy that I could try?

-- Thanks
-- Darren --

Two things, have you tried upgrading to the latest version, 6.0? And how do
you read the initial document and the files?

--

Joe Fawcett (MVP - XML)

http://joe.fawcett.name

"Darren Morby" <DarrenMorby@discussions.microsoft.com> wrote in message
news:07584BA0-4CB3-47B6-B020-048C072FB29A@microsoft.com...
> Hi, I'm using MSXML4 and I'm having an issue with memory allocation which
> appears to be a memory leak, but isn't exactly. Let me explain.
>
> I have this one XML file (the "IXM" file) that contains the names of 148
> other XML files (the "external" files). What is supposed to happen is that
> the app reads the IXM file, then reads each external file named in the IXM
> file, then copies the external file's data to the database. The problem
> is
> that after about 36 external files, it runs out of memory. Each external
> file
> is generally between 104 MB and 106 MB, but some are larger. The largest
> is
> 136 MB. The external files are in ASCII.
>
> I run performance monitor and monitor virtual bytes. It seems that the
> virtual bytes are increasing all the time, and by the time of the 36th
> external file, virtual bytes tries to exceed 2 GB, and it bombs.
>
> Based on experiments I've done, I conclude the following:
>
> 1. When MSXML is allocating memory, it is not doing so on the process
> heap,
> but on its own heap, so memory looks like this:
>
> [PROCESS-HEAP][HEAP-1][HEAP-2][MSXML-HEAP-1][MSXML-HEAP-2]
>
> 2. The SAX parser in MSXML reads the entire external file into a
> contiguous
> region of memory before it will call the content handler.
>
> 3. If MSXML cannot load the entire external file into a contiguous region
> of
> memory in its own heaps, it allocates another heap:
>
> [PROCESS-HEAP][HEAP-1][HEAP-2][MSXML-HEAP-1][MSXML-HEAP-2][MSXML-HEAP-3]
>
> [PROCESS-HEAP][HEAP-1][HEAP-2][MSXML-HEAP-1][MSXML-HEAP-2][MSXML-HEAP-3][MSXML-HEAP-4]
>
> [PROCESS-HEAP][HEAP-1][HEAP-2][MSXML-HEAP-1][MSXML-HEAP-2][MSXML-HEAP-3][MSXML-HEAP-4][MSXML-HEAP-5]
>
> It does not seem to be freeing the extra heaps that it is creating. So
> eventually, it will try to allocate a memory block that's too big for the
> existing heaps and too big to allocate a new heap.
>
> Assuming that's the case:
>
> 1. How do I free up all the heaps that MSXML allocates, preferably between
> external files?
>
> 2. How do I get MSXML to use the app's heap instead of creating its own?
>
> 3. Is there any other strategy that I could try?
>
> -- Thanks
> -- Darren --

I have MSXML 6 installed, but I don't think I'm using it. I also don't know
if I'm supposed to be supporting MSXML 6 and not MSXML 4.

I created a wrapper class that wraps the MFC CFile (open on the actual input
file) with the ISequentialStream interface. This object is passed as the
parameter to the "parse" method of ISAXXMLReader. I'm using the
SAXXMLReader40 coclass, so I guess that I'm not using MSXML 6.

--
-- Darren --


"Joe Fawcett" wrote:

> Two things, have you tried upgrading to the latest version, 6.0? And how do
> you read the initial document and the files?
>
> --
>
> Joe Fawcett (MVP - XML)
>
> http://joe.fawcett.name
>
> "Darren Morby" <DarrenMorby@discussions.microsoft.com> wrote in message
> news:07584BA0-4CB3-47B6-B020-048C072FB29A@microsoft.com...
> > Hi, I'm using MSXML4 and I'm having an issue with memory allocation which
> > appears to be a memory leak, but isn't exactly. Let me explain.
> >
> > I have this one XML file (the "IXM" file) that contains the names of 148
> > other XML files (the "external" files). What is supposed to happen is that
> > the app reads the IXM file, then reads each external file named in the IXM
> > file, then copies the external file's data to the database. The problem
> > is
> > that after about 36 external files, it runs out of memory. Each external
> > file
> > is generally between 104 MB and 106 MB, but some are larger. The largest
> > is
> > 136 MB. The external files are in ASCII.
> >
> > I run performance monitor and monitor virtual bytes. It seems that the
> > virtual bytes are increasing all the time, and by the time of the 36th
> > external file, virtual bytes tries to exceed 2 GB, and it bombs.
> >
> > Based on experiments I've done, I conclude the following:
> >
> > 1. When MSXML is allocating memory, it is not doing so on the process
> > heap,
> > but on its own heap, so memory looks like this:
> >
> > [PROCESS-HEAP][HEAP-1][HEAP-2][MSXML-HEAP-1][MSXML-HEAP-2]
> >
> > 2. The SAX parser in MSXML reads the entire external file into a
> > contiguous
> > region of memory before it will call the content handler.
> >
> > 3. If MSXML cannot load the entire external file into a contiguous region
> > of
> > memory in its own heaps, it allocates another heap:
> >
> > [PROCESS-HEAP][HEAP-1][HEAP-2][MSXML-HEAP-1][MSXML-HEAP-2][MSXML-HEAP-3]
> >
> > [PROCESS-HEAP][HEAP-1][HEAP-2][MSXML-HEAP-1][MSXML-HEAP-2][MSXML-HEAP-3][MSXML-HEAP-4]
> >
> > [PROCESS-HEAP][HEAP-1][HEAP-2][MSXML-HEAP-1][MSXML-HEAP-2][MSXML-HEAP-3][MSXML-HEAP-4][MSXML-HEAP-5]
> >
> > It does not seem to be freeing the extra heaps that it is creating. So
> > eventually, it will try to allocate a memory block that's too big for the
> > existing heaps and too big to allocate a new heap.
> >
> > Assuming that's the case:
> >
> > 1. How do I free up all the heaps that MSXML allocates, preferably between
> > external files?
> >
> > 2. How do I get MSXML to use the app's heap instead of creating its own?
> >
> > 3. Is there any other strategy that I could try?
> >
> > -- Thanks
> > -- Darren --
>
>
>

I just found out. I have to support MSXML 4.0 SP2.
I had written:
> I have MSXML 6 installed, but I don't think I'm using it. I also don't know
> if I'm supposed to be supporting MSXML 6 and not MSXML 4.

-- Darren --

"Darren Morby" <DarrenMorby@discussions.microsoft.com> wrote in message
news:8A3D9324-5517-4DBB-AD3A-60F1879EA304@microsoft.com...
>I just found out. I have to support MSXML 4.0 SP2.
> I had written:
>> I have MSXML 6 installed, but I don't think I'm using it. I also don't
>> know
>> if I'm supposed to be supporting MSXML 6 and not MSXML 4.
>

That could be a problem for you. If this turns out to be a real MSXML4 bug
you'll have trouble getting a fix. MS have continued with supporting MSXML3
and 6 but are actively discouraging the use of 4 and 5.

This KB may be worth a read:-

http://support.microsoft.com/kb/304227


--
Anthony Jones - MVP ASP/ASP.NET

Actually, I also found this page to be useful:
http://msdn.microsoft.com/en-us/xml/bb291077.aspx

I will try to see if MSXML 6 is better behaved memory-wise, and then try to
work up the chain of command to get it used instead of MSXML 4.

-- Darren --

"Anthony Jones" wrote:
> That could be a problem for you. If this turns out to be a real MSXML4 bug
> you'll have trouble getting a fix. MS have continued with supporting MSXML3
> and 6 but are actively discouraging the use of 4 and 5.