plot of large data file = large .eps file? - Graphics

On 2006-01-06, tv <tommaso.vinci@gmail.com> wrote:
>
> It's because you need the libpdf to generate pdf...
> try to look with apt

Unfortunately, I don't know what Debian package I could be
missing here... If i search with apt, i get:

mc303@forsythe:~$ apt-cache search libpdf
libpdf-api2-perl - provides the functions for creating PDF documents in Perl
libpdf-report-perl - simplified PDF report generator
libpdflib804 - [Physics] Comprehensive library of parton density functions
libpdflib804-dev - [Physics] Comprehensive library of parton density functions

but I guess it's neither of these that i need to get the pdf terminal in
gnuplot :-(

> and re-build gnuplot from source, try the 4.1
> version it has a lot of awesome new features

I am only a user, not the administrator of my machine... I think I would like
to stick to the version that's installed on my machine without having to
compile and install it in my own home-directory...

The version I have available is

G N U P L O T
Version 4.0 patchlevel 0
last modified Thu Apr 15 14:44:22 CEST 2004
System: Linux 2.6.8-2-686-smp

and `show version long' gives me:

Compile options:
+READLINE -LIBREADLINE +GD_PNG +GD_JPEG +GD_TTF -PNG
+NOCWDRC +X11 +USE_MOUSE

DRIVER_DIR = "/usr/lib/gnuplot"
HELPFILE = "/usr/share/gnuplot/gnuplot.gih"
CONTACT = <XXXXXXXXXXXXX@lists.sourceforge.net>
HELPMAIL = <XXXXXXXXXXXXX@lists.sourceforge.net>

`set term' tells me I have the 'png' terminal available (and PNG images do
work) but there's no `pdf' terminal in there. Could anybody tell me what
Debian packages I have to ask our sysadmins to install so we can have the
`pdf' terminal also?

> you'll certainly need at least:
> libpdf to generate pdf

What (stable) Debian package would be needed here? I can't seem to find
anything usefull with `apt-cache search libpdf'...

> and libgd to generate all the graphics: png jpeg etc... (now you can
> directly generate from gnuplot animated gif pictures)

This is what is on my system:

bartv@vonneumann:~$ dpkg --list | grep libgd
ii libgd-gd2-perl 2.23-1 Perl module wrapper for libgd - gd2 variant
ii libgd-tools 2.0.33-1.1 GD command line tools and example code
ii libgd2-xpm 2.0.33-1.1 GD Graphics Library version 2
rc libgda2-1 1.0.4-3 GNOME Data Access library for GNOME2
ii libgda2-3 1.2.1-2sarge1 GNOME Data Access library for GNOME2
ii libgda2-common 1.2.1-2sarge1 Common files for GNOME Data Access library f
ii libgdbm3 1.8.3-2 GNU dbm database routines (runtime version)
ii libgdk-pixbuf2 0.22.0-8.1 The GdkPixBuf image library, gtk+ 1.2 versio
ii libgdkcardimag 2.6.2-3 drawing library for card games (runtime)

generating PNG images works.

> you probably have a version of those libraries ready with apt-get

If anybody could tell me what (stable) Debian package needs to be installed
in order to have the `pdf' terminal available, that would be nice because then
I could ask our sysadmins to install it.

Thanks,
Bart

--
"Share what you know. Learn what you don't."

Bart Vandewoestyne ha scritto:

> On 2006-01-06, tv <tommaso.vinci@gmail.com> wrote:

sorry.. my fault... it's called pdflib.... not libpdf...

if they're not on your system, you can download from
http://www.pdflib.com/products/pdfli...e-6.0.2.tar.gz

and install in your own directory, there's no problem at all....
moreover you can
rebuild in your own directory gnuplot and use it from there, there's
no problem at all for this task... even without root access, of course

> ii libgd2-xpm 2.0.33-1.1 GD Graphics Library version 2

now you have gd and it's a new version so you don't need to rebuild
those!

just make sure that when you recompile gnuplo to tell him where pdflib
are (of course if they're already installed you don't need to rebuild
those)

to configure gnuplot you'll find a huge amount of settings you can use:

../configure --with-readline=gnu --enable-history-file

are IMHO very useful you'll find history and tab-completion...

good luck

In article <1136540283.830350@seven.kulnet.kuleuven.ac.be>,
Bart Vandewoestyne <MyFirstName.MyLastName@telenet.be> wrote:
>On 2006-01-06, Ethan Merritt <merritt@u.washington.edu> wrote:
>>
>> \usepackage{graphicx}
>> ...
>> \includegraphics[clip]{figure.png}
>>
>>>I did some quick googling which learned me
>>>that i might have to use `pdflatex' instead of `latex' then?
>>
>> Not that I know of. I don't normally use pdflatex.
>
>Hmm... so the above works for you if you simply compile your
>document with `latex myfile.tex'? Strange... if I do this, then
>i get:
>
>LaTeX Error: Cannot determine size of graphic in mydata.png (no BoundingBox)
>
>If i process it with `pdflatex file.tex' then it works...

I will have to leave it to TeX/LaTeX gurus to explain how this
all works. I basically only know how to drive LaTeX via the
interactive front-end LyX. Whatever LyX does internally, works.
I know it doesn't requite pdflatex because that's an optional
package, and I didn't select that option. Beyond that, ...
I just don't know.
--
Ethan A Merritt

On 2006-01-05, Ethan Merritt <merritt@u.washington.edu> wrote:
>
>> The problem is that due to the one-million data-points, the size
>> of the .eps file grows up to 7.1 MB. And I have to include
>> several of these into my LaTeX document, which makes the size of
>> my LaTeX document quite large and also reading it with some
>> PostScript viewer is not a pleasant experience because the
>> figures take quite a while to load.
>
> PostScript (including eps) is a vector-based description of
> your plot, and contains an explicit description of every one
> of your millions of points even though most of them lie on
> top of each other and cannot be seen.
>
> [...]
>
> As you have already discovered, you can also down-sample
> your data set to reduce the size of an *.eps image.
> There is, however, a strong disadvantage to doing this in
> some cases. If the purpose of your plot is to discover and
> highlight outliers or minority regions in the distribution
> of points, then by down-sampling you may lose exactly the
> information you want to highlight. A pixel-based rendering
> of all points does not suffer from this effect.
>
> [...]
>
> If the small features are just as important (or more important)
> than the large features, then I recommend using png rather than
> eps output for the reasons I gave above.

I think I have found quite a nice solution that still allows me
to use an EPS file for my output, and I don't loose any graphical
information at all. Because I think this might be of interest
for many gnuplot users having large data-files, this is how i do
it:

* I resample my data-file so i get a smaller one in size. The
resampling is based on the resolution that i want in my .eps
file (say 100dpi, 300dpi or so...)
* I then simply plot the reduced data file. The output is quasi
identical to the .eps with the full data file, but much smaller
in size!

The datafile I'm testing it on is 16MB and can be downloaded from:

http://www.cs.kuleuven.ac.be/~bartv/...lem/mydata.dat

The gnuplot file I'm using to plot this data is at:

http://www.cs.kuleuven.ac.be/~bartv/...blem/myplot.pl

If i plot the full data file, I get an .eps file of 14MB:

http://www.cs.kuleuven.ac.be/~bartv/...lem/mydata.eps

I then downsample my data-file so that i only keep values for
which the dots representing the (X,Y)-pairs will not overlap in
my .eps file. Downsampling happens with this (quite simple) AWK-script:

http://www.cs.kuleuven.ac.be/~bartv/...lem/reduce.awk

After downsampling to 300dpi with the command invocation

$ ./reduce.awk mydata.dat 300 0.5 0.5 1 1e6 1e-20 1 > mydata_reduced.dat

the reduced data file is only 393KB:

http://www.cs.kuleuven.ac.be/~bartv/...ta_reduced.dat

and the resulting .eps file is then only 338KB:

http://www.cs.kuleuven.ac.be/~bartv/...ta_reduced.eps

So the resulting .eps is only 2.37 percent of the size of the
..eps produced with the full data!!! Quite a nice compression we
have there :-)

There is only one comment i could make, and that is that for now,
i take the standard width and height of a gnuplot enhanced eps
picture to be 360x252 PostScript points. Actually, that is not
correct because this is the size of the *full* eps file,
including the X and Y labels etc. If i could figure out a way to
get the size in PostScript points of the *actual* rectangle where
the data is plotted, then I would even be able to get an even
larger reduction in size. But I don't know if it's possible to
find out the size of the actual drawing area... and I'm even not
sure if its size can vary depending on the fonts of the labels
etc...

As I believe my AWK script might be of use for quite a lot of the
users here, I kindly would ask the readers here to send me any of
your comments and suggestions. Especially on bugs, programming
style, possible improvements in either functionality or speed etc...

Regards,
Bart

--
"Share what you know. Learn what you don't."

In article <1137100105.821583@seven.kulnet.kuleuven.ac.be>,
Bart Vandewoestyne <MyFirstName.MyLastName@telenet.be> wrote:
>
>So the resulting .eps is only 2.37 percent of the size of the
>.eps produced with the full data!!! Quite a nice compression we
>have there :-)
>
>There is only one comment i could make, and that is that for now,
>i take the standard width and height of a gnuplot enhanced eps
>picture to be 360x252 PostScript points. Actually, that is not
>correct because this is the size of the *full* eps file,
>including the X and Y labels etc. If i could figure out a way to
>get the size in PostScript points of the *actual* rectangle where
>the data is plotted, then I would even be able to get an even
>larger reduction in size.

Most eps viewing program show you the x,y coordinates of the
cursor position in PostScript coordinates. So just open up
your eps file in ghostview or gv or whatever, and read off the
coordinates of the plot corners.

I still think it would be a whole lot easier just to use png
format.
--
Ethan A Merritt

On 2006-01-12, Ethan Merritt <merritt@u.washington.edu> wrote:
>
> I still think it would be a whole lot easier just to use png
> format.

Why? My solution is there and apparently it's working :-)

As for the PNG format... I've tried that, but i was a bit
disappointed about the quality... and the fonts I tried for the
labels did quite strange things or looked not as I wanted them to
look... even with the standard 'Computer Modern Roman' that LaTeX uses :-(

With my resampling-solution, I again have my scalable EPS figure
and I can use all fonts and sizes that I'm used to for EPS. Next
to that, I can still use the 'latex' command instead of
'pdflatex' so i don't have to convert all my other .eps files to
..png or .pdf or so, which is also quite nice.

Best wishes,
Bart

--
"Share what you know. Learn what you don't."