The Harrington Compression Method (HCM) White Paper.

Ok updates:

1) Run it on files 1, 2, and 3. A modeling error was found with just
doing it on 1. This is not an issue afterwards.

2) Statistician was total waste of time. She saw binary and could not
keep up.

3) A computer science Doctorate gave me some time. He thumped Counting
argument nonstop, I thumped 'returns original data' each time and
'covers all possible outcomes'. He is not convinced but will give me a
chance to do a full on written step by step process, including
variations possible and how it conquers them at each stage.

His argument was very persuasive. He knows his stuff. Counting does
seem to be an issue, yet... I know for a fact all outcomes can
compress... I did model after model. But I did have a modeling error
in the whitepaper, though that was a typo in my excel sheets more than
anything else (the rest has no typo, verified while I was there).


So Earl I guess you are going to say: B

However I am not saying B, I am going to examine the code, make some
suggestions if needed, and proceed.

I am so tired to, today I got 4 hours sleep, and yesterday 5 hours.
And I have been physically active at work. So I might run late
tommorrow, I apologize for 'keeping you all in suspense'.



Hmmm, actually I have had enough time to examine some of the code, yes
you are missing a couple of steps, this will be easy to correct
however. But I have 2 minutes before I have to leave for work, and
therefore am unable to give instructions now. However expect some
comments in the morning.

Nice work btw, my thanks sir. The fixes will be very good to see

On Aug 26, 11:24*pm, Einstein <michae...@gmail.com> wrote:
> Please try to at least act like your trying.

I *did* try, successfully as it turns out, fixing the bugs in your
method some 15 years ago! (I found an easy way to locate my patents
on Internet: *another* James D. Allen, in Florida, has incorporated
mine on his webpage):
http://www.patentgenius.com/inventor/AllenJamesD.html

(The patent covers a method to interleave fixed
var-length-to-var-length codes with *no* overhead for
control or steering bits. Such control bits are
avoided automatically in arithmetic coding, but why
waste time or silicon on an arithmetic coder if you
don't need it?)


I find minimal fault with amateur mathematicians who try
to "square the circle" -- the impossibility of this, IIRC,
was not proven until the 19th century. Nor do I fault
those who seek Perpetual Motion Machines -- that
"impossibility" proof depends on assumptions about the nature
of time which are increasingly seen as controversial.

Yet the "counting argument" against Perpetual Compression
doesn't require any graduate-level mathematics or controversial
philosophy to understand. Instead I should expect any
intelligent teenager to be able to comprehend it.

When someone posts a cute math proof that 2+2 = 5, it is
sometimes difficult to find the precise bug. But we are
nevertheless sure there is one!

This does not mean that Proofs of 2+2 = 5, in alt.math.rec
will be ignored: some may find it an interesting challenge,
especially if the bug is well hidden. No one, however,
reads such an article worried that one's lifelong view of
arithmetic is about to be upset. I'm sure I can speak for
other comp.compression'ers in saying that any attention paid
to this thread would be similar to that of a "2+2 = 5" thread.

Einstein wrote:
> This post is to help people understand for two purposes,
> Investment in the proposal, ...

You may very well succeed there! Make some sexy Power-Point
slides, hire a few studious-looking types, and build a nice
software compress-anything demo. The VC's are unlikely to
study your source code in any detail, so software development
should focus on a sexy user interface, with only a trivial
pass-through needed for the compress/decompress demo.

I would underplay the idea that you can compress arbitrary
"entropic" data -- VC's will have above-average math skill --
and instead come up with some mumbo-jumbo about your method
uncovering otherwise-hidden "patterns".

* * * * * * * * * * * * *

No one expects a debugged program when an algorithm is
first described, but it should still be stated clearly and
concisely. Recently I asked my sister for a recipe and,
after stating the ingredients she wrote:
> Mix first 5 ingredients thoroughly
> Stir in remaining ingredients
> Cover and chill
> Makes 5 cups
Short and to the point! No long paragraphs about whether
this potato salad is better than Aunt Judy's potato salad.

I'm afraid that you, Einstein, have combined your method
description with your advertising hype. Try again, pretending
that you're telling a youngster of average intelligence
*exactly* what to do.

I've stripped your "method" description of hype and irrelevancy
leaving what seem to be the essential step descriptions:

> 1)[Start with] a standard Huffman table:
>
> 2)Using HCM, we make different 'files' out of information. Any 1's in
> a line (until three are used or a 0 occurs) will lead to a different
> 'file' being granted the following information from that string.
>
> 3)Repeat step #2 on the #1 files content.

What does step #2 mean? What is a "line"?
If a line happens to be 111000101010, does that mean
that only 000101010 ("the following information") is
sent to the "different 'file'"? If so, you seem
to assume the data is *not* "entropic". If the 111
is also to be sent to the "different 'file'", your
decompressor will not be able to reconstruct compressor's
decision.

I'm not trying to be difficult; I honestly can't follow
your description. Lead us through it, slowly, with simple
examples (and no unnecessary hype).

If made rigorous, I suspect your method *would* work,
but only on non-"entropic" data, and only as well as
a myriad of other existing methods. When you address the
details, you may even end up with a design *I* came up with
15 years ago or so. I have a patent on it! Maybe you'll
need to share profits with me. :-) :-)

> .... Eventually there is a point where the fog is so
> thick that any forward movement will not gain you any
> viewing distance, and even eventually the fog grows so
> thick you cannot see at all.

I'll ponder this: perhaps it might be a good metaphor
for understanding the HCM method!

James Hussein Allen

On Aug 28, 6:00 am, Einstein <michae...@gmail.com> wrote:
> Ok updates:
>
> 1) Run it on files 1, 2, and 3. A modeling error was found with just
> doing it on 1. This is not an issue afterwards.
>
> 2) Statistician was total waste of time. She saw binary and could not
> keep up.
>
> 3) A computer science Doctorate gave me some time. He thumped Counting
> argument nonstop, I thumped 'returns original data' each time and
> 'covers all possible outcomes'. He is not convinced but will give me a
> chance to do a full on written step by step process, including
> variations possible and how it conquers them at each stage.
>
> His argument was very persuasive. He knows his stuff. Counting does
> seem to be an issue, yet... I know for a fact all outcomes can
> compress... I did model after model. But I did have a modeling error
> in the whitepaper, though that was a typo in my excel sheets more than
> anything else (the rest has no typo, verified while I was there).
>
> So Earl I guess you are going to say: B
>
> However I am not saying B, I am going to examine the code, make some
> suggestions if needed, and proceed.
>
> I am so tired to, today I got 4 hours sleep, and yesterday 5 hours.
> And I have been physically active at work. So I might run late
> tommorrow, I apologize for 'keeping you all in suspense'.
>
> Hmmm, actually I have had enough time to examine some of the code, yes
> you are missing a couple of steps, this will be easy to correct
> however. But I have 2 minutes before I have to leave for work, and
> therefore am unable to give instructions now. However expect some
> comments in the morning.
>
> Nice work btw, my thanks sir. The fixes will be very good to see

James Hussein Allen wrote:
> > 1)[Start with] a standard Huffman table:
>
> > 2)Using HCM, we make different 'files' out of information. Any 1's in
> > a line (until three are used or a 0 occurs) will lead to a different
> > 'file' being granted the following information from that string.
>
> > 3)Repeat step #2 on the #1 files content.
>
> What does step #2 mean? *What is a "line"?
> If a line happens to be 111000101010, does that mean
> that only 000101010 ("the following information") is
> sent to the "different 'file'"? *If so, you seem
> to assume the data is *not* "entropic". *If the 111
> is also to be sent to the "different 'file'", your
> decompressor will not be able to reconstruct compressor's
> decision.
>
> I'm not trying to be difficult; I honestly can't follow
> your description. *Lead us through it, slowly, with simple
> examples (and no unnecessary hype).

The understanding that I got, from examples and other people's
examples:
As you perform the Huffman encoding on the original string, right each
translated symbol vertically. For instance, 10 00 01 11 01 would
become
1 0 1 1 1
1 _ 0 1 0
0 _ _ 1 _
If we then read the lines horizontally, ignoring any gaps, we get the
three files: 10111, 1010 and 01. Assuming that all 00, 01, 10, and 11
are evenly distributed, we'll get 3/4 of the bits in the first file as
1, 2/3 in the second, and 1/2 in the third. This matches the
proportions he predicts, at least for files 2 & 3 (he gives figures
for splitting file 1 into three, which I don't), so I suspect this
interpretation is correct. It also yields the same results as the
sample he provides in the original post.

One detail that's buried at the bottom of the original post is talk
about modifying the second Huffman table for different situations. He
doesn't make it clear, however, when these modifications should be
made, or what they should be. I suspect that the missing step will be
related to this.

Mark Sherry

On Aug 28, 6:00 am, Einstein <michae...@gmail.com> wrote:
> 1) Run it on files 1, 2, and 3. A modeling error was found with just
> doing it on 1. This is not an issue afterwards.

Translation: I messed up. My models were wrong and I spent a lot of
time insulting people that they were to dumb to follow my logic. Now
I will refuse to admit that I was the one as fault from the start.

> 2) Statistician was total waste of time. She saw binary and could not
> keep up.

Translation: She knew too much math for me to snow her under with big
words. After the fact that all information on modern computers is
recorded in one binary format or another means nothing is I say
otherwise.

On Aug 28, 6:00 am, Einstein <michae...@gmail.com> wrote:
> 3) A computer science Doctorate gave me some time. He thumped Counting
> argument nonstop, I thumped 'returns original data' each time and
> 'covers all possible outcomes'.

Transalation: Another person who refused to be snowed over on just my
say-so. After-all just because I don't have any proof does not mean
he should not just take my word as fact!

> He is not convinced but will give me a
> chance to do a full on written step by step process, including
> variations possible and how it conquers them at each stage.

Which raises to interesting points.

1) Why is it you can go thru *ALL* the steps with this person but to
date have refuse to do so in this Newsgroup.

2) You claimed that you went thru *all* the steps but he still was not
convinced, this suggests you did not get convincing results yourself.
Hummm, I wonder why?

> His argument was very persuasive. He knows his stuff. Counting does
> seem to be an issue, yet... I know for a fact all outcomes can
> compress...

How do you know it is a fact when you refuse to do a complete set of
tests?

> I did model after model. But I did have a modeling error
> in the whitepaper, though that was a typo in my excel sheets more than
> anything else (the rest has no typo, verified while I was there).

Interesting you did model after model and still end up publishing a
bad model in your 'whitepaper'? Most people would realize that is a
clear sign there is something wrong with their models.

On Aug 28, 6:00 am, Einstein <michae...@gmail.com> wrote:
> So Earl I guess you are going to say: B

> However I am not saying B, I am going to examine the code, make some
> suggestions if needed, and proceed.

In other words you are trying to shift all the blame on the
programmer, and avoid facing the fact that:

1) You still to date have not *CLEARLY* described your system. How do
you think the programmer could mess up if you were not so vague about
a number of points.

2) Refused to do a step by step example from start to finish. If you
can do it for others, you can do it here too.

3) Refused to listen to suggestions from people (not me, I am just a
'dumb' programmer) with not only more experience with new compression
systems than you, but people who in-fact have gone the entire
development cycle of new compression methods and delivered working
results.

On Aug 28, 6:00 am, Einstein <michae...@gmail.com> wrote:
> I am so tired to, today I got 4 hours sleep, and yesterday 5 hours.
> And I have been physically active at work. So I might run late
> tommorrow, I apologize for 'keeping you all in suspense'.

> Hmmm, actually I have had enough time to examine some of the code, yes
> you are missing a couple of steps, this will be easy to correct
> however. But I have 2 minutes before I have to leave for work, and
> therefore am unable to give instructions now. However expect some
> comments in the morning.

This is even worse. If you are tired get some sleep first. If you
try to debug code while having so little sleep you are bound to just
make things worse, not better. Sleep well, sit down and think
carefully about what your system is suppose to do (take notes, it
usually helps) then look at the code. Hope to see you changes over
the week-end.

Earl Colby Pottinger

On Aug 28, 8:00*am, Einstein <michae...@gmail.com> wrote:
> 3) A computer science Doctorate gave me some time. He thumped Counting
> argument nonstop, I thumped 'returns original data' each time and
> 'covers all possible outcomes'.

As scientist also this newsgroup is full of mantra speakers they can
better point out where you go wrong.

I think you go wrong in your calculations, because the first file/
stream is indeed smaller then the original minus the gain you gave
with the first conversion but the second and third files/streams are
already bigger then the gain of the first.

Let say we take 1,000,000 bits random input
Then after converting the bits we get 1,124,714 bits
After sorting and converting again we get:
file/stream1: 500,000 bits (converted 422,106 bits -5.05%)
file/stream2: 374,754 bits (converted 353,682 bits +6.14)
file/stream3: 249,960 bits (not conv. 249,960 bits +11.12%)

422,106 + 353,682 + 249,960 = 1,025,748 bits
So a gain of 25,748 bits (+2.57%) after one pass.

To the software maker


Steps in short:

1) String length source file, if odd numbered either trunc, or add a
bit [command+1]
2) COUNT all 11, 10, 01, 00 outcomes. Switch them into most occurring
to most desired (In this order desired: 11, 10, 01, 00). [Command
+8]
3) 1st Huffman
4) File Sort
5) String length files, if odd numbered either trunc, or add a bit
[command+3]
6) COUNT all 11, 10, 01, 00 outcomes per file. Switch them into most
occurring to most desired (In this order desired: 11, 10, 01, 00).
[Command +24]
7) repeat 1st Huffman on all 3 files.
8) Subfile sort.
9) String length sub-files, if odd numbered either trunc, or add a
bit [command+9]
10) COUNT all 11, 10, 01, 00 outcomes per file. Switch them into most
occurring to most desired (In this order desired: 11, 10, 01, 00).
[Command +24]
11) Compression Huffman on files 1.1, 1.2, 1.3, 2.1, 2.2, 2.3, 3.1,
and 3.2. Do not attempt file #3 unless COUNT indicated 11 results >
01+11 results.
12) String Length each subfile, use 50 bits per subfile to identify
string length [command+450]
13) Append information as:
CommandSection&File1.1&File1.2&File1.3&File2.1&Fil e2.2&File2.3&File3.1&File3.2&File3.3



To the rest...

I can read your posts in due time, but I am still waking up. I am
however making one post in a new thread on purpose...