request comments: simple c program builder in lisp

> (defun concat-string (&rest args)
> (apply #'concatenate args))
>

hum this should be

(defun concat-string (&rest args)
(apply #'concatenate 'string args))

Well it should not be at all really !

Sacha

På Sun, 15 Jun 2008 10:09:08 +0200, skrev Sacha <none@address.spam>:

>
>> (defmacro concat-str (&rest args)
>> `(concatenate 'string ,@args))
>>
>
> You don't need a macro for this :
>
> (defun concat-string (&rest args)
> (apply #'concatenate args))
>

Right!

> But :
>
> ;CL-USER 12 > CALL-ARGUMENTS-LIMIT
> ;2047
>
> So on my lisp implementation, your macro (my function as well) would
> break for lists with over 2047 items. besides we're not quite sure on
> the performance implications of such a big parameter list.
>
> So :
>
> (defun list->stream (list stream)
> (loop for item in list
> do (princ item stream)))
>

Do you think you would ever want to concatenate more that 2048 elements in
one line?

(defun concat-str (&rest list)
(apply #'concatenate 'string list))

is fine here.

For printing a list with a delimiter how about format?

CL-USER 3 > (format nil "~{~A~^ ~}" (list 1 2 3 4 5))
"1 2 3 4 5"

or

CL-USER 17 > (defun list->string (list &optional (delim " "))
(format nil (concat-str "~{~A~^" delim "~}") list))
LIST->STRING

CL-USER 18 > (list->string (list 1 2 3 4 5))
"1 2 3 4 5"

CL-USER 19 > (list->string (list 1 2 3 4 5) ", ")
"1, 2, 3, 4, 5"

--------------
John Thingstad

John Thingstad wrote:

> Do you think you would ever want to concatenate more that 2048 elements
> in one line?
>
> (defun concat-str (&rest list)
> (apply #'concatenate 'string list))
>
> is fine here.

True, but there's more to it !
As you nest more and more functions (let's say for html generation),
you'll concatenate smaller strings, then concatenate these again at a
higher level, and so on up to the document level. That's a lot of
useless consing. Your processor has to move the same characters again
and again.

A cost which isn't alleviated by easier programming or readability.

In the end odds are that you still want to print all this to a stream,
so why all the consing ?

Also you can't concatenate strings to characters and numbers, you're
loosing expressiveness.


> For printing a list with a delimiter how about format?
>
> CL-USER 3 > (format nil "~{~A~^ ~}" (list 1 2 3 4 5))
> "1 2 3 4 5"

True enough, the tool is there and deserve to be used.

Then again when scaling to large nested trees/function calls this won't
make it. And this time we're loosing readability. I tend to use format
only for the lower levels.

Anyways that's a matter of taste, to each his own style =)

Sacha

<parth.malwankar@gmail.com> wrote:
+---------------
| I am fairly new to Lisp and am trying to write a simple build tool
| for C files in order to learn lisp. I am not really interested in any
| advanced features for the build system yet as the main intent is to
| learn Lisp.
+---------------

Understood, and *welcome*!

That said, at some point [though probably not now!] you should
take a look at the various packages/libraries listed here:

http://www.cliki.net/development

and especially at these:

http://www.cliki.net/asdf [currently quite popular]
Another System Definition Facility

http://www.cliki.net/mk-defsystem [much older, but still used by some]
MK-defsystem is a system definition utility; it fills a similar
role for CL Development as make(1) does for C. ...

Note that ASDF (and possibly MK-DEFSYSTEM?) already has some hooks for
C source files, though you will have to define your own specializations
of the PERFORM method for the COMPILE-OP and LOAD-OP operations, as it
tends to be very application- and/or platform- and/or implementation-
dependent. [E.g. What do you want to *do* with your compiled C code?
Run it? Build a DSO & link to it?] But several of the readily-available
CL libraries have useful examples in their ".asd" files:

http://www.cliki.net/Osicat
http://www.cliki.net/Linedit
http://www.cliki.net/db-sockets
http://www.cliki.net/CL-Ncurses

They're all pretty similar in their handling of C files. The main
thing you might have to do is tweak the RUN-SHELL-COMMAND calls
[and/or extend the definition of RUN-SHELL-COMMAND in "asdf.lisp"
if your platform or CL implementation isn't supported].


-Rob

-----
Rob Warnock <rpw3@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607

On 15 Giu, 12:53, r...@rpw3.org (Rob Warnock) wrote:
> <parth.malwan...@gmail.com> wrote:
>
> +---------------
> | I am fairly new to Lisp and am trying to write a simple build tool
> | for C files in order to learn lisp. I am not really interested in any
> | advanced features for the build system yet as the main intent is to
> | learn Lisp.
> +---------------
>
> Understood, and *welcome*!
>
> That said, at some point [though probably not now!] you should
> take a look at the various packages/libraries listed here:
>
> http://www.cliki.net/development
>
> and especially at these:
>
> http://www.cliki.net/asdf [currently quite popular]
> Another System Definition Facility
>
> http://www.cliki.net/mk-defsystem [much older, but still used by some]
> MK-defsystem is a system definition utility; it fills a similar
> role for CL Development as make(1) does for C. ...
>
> Note that ASDF (and possibly MK-DEFSYSTEM?) already has some hooks for
> C source files,

MKEFSYSTEM does C and Fortran out of the box. AFAIK Matlisp is
still compiled from scratch with MKEFSYSTEM as it needs to compile
largish Fortran libraries.

Cheers
--
Marco

On Jun 15, 3:01 pm, "John Thingstad" <jpth...@online.no> wrote:
> På Sun, 15 Jun 2008 10:09:08 +0200, skrev Sacha <n...@address.spam>:
>
>
>
> >> (defmacro concat-str (&rest args)
> >> `(concatenate 'string ,@args))
>
> > You don't need a macro for this :
>
> > (defun concat-string (&rest args)
> > (apply #'concatenate args))
>
> Right!
>
> > But :
>
> > ;CL-USER 12 > CALL-ARGUMENTS-LIMIT
> > ;2047
>
> > So on my lisp implementation, your macro (my function as well) would
> > break for lists with over 2047 items. besides we're not quite sure on
> > the performance implications of such a big parameter list.
>
> > So :
>
> > (defun list->stream (list stream)
> > (loop for item in list
> > do (princ item stream)))
>
> Do you think you would ever want to concatenate more that 2048 elements in
> one line?
>
> (defun concat-str (&rest list)
> (apply #'concatenate 'string list))
>
> is fine here.
>
> For printing a list with a delimiter how about format?
>
> CL-USER 3 > (format nil "~{~A~^ ~}" (list 1 2 3 4 5))
> "1 2 3 4 5"
>
> or
>
> CL-USER 17 > (defun list->string (list &optional (delim " "))
> (format nil (concat-str "~{~A~^" delim "~}") list))
> LIST->STRING
>
> CL-USER 18 > (list->string (list 1 2 3 4 5))
> "1 2 3 4 5"
>
> CL-USER 19 > (list->string (list 1 2 3 4 5) ", ")
> "1, 2, 3, 4, 5"
>
> --------------
> John Thingstad

Thanks everyone for your comments.
This really helped me understand the idiomatic lisp
programming. I did some more updates based on the comments.

- Now I am using using the approach of keeping items in
list format as much as possible and creating a string
only in the end (lst->str).
This was quite an interesting approach which didn't
really occur to me possibly due to my background in
languages like C and Python. I assume this will be
much more efficient than what I was doing earlier.
Thanks Sacha.

- Using format for concatenation the list is very cool.
I knew about this but some how it didn't occur to me
... probably need more practice

- There is also the 'emit' and 'emit-to-screen' method
that allows the user to choose between actually building
the c code vs just printing the output on screen.

- I also got a chance to add a 'clean' method that deletes
all the generated files during the build process.

Here is a simple interaction in clisp:

[1]> (load "packages.lisp")
.............. snip ................................
T
[2]> (load "bld.lisp")
T
[3]> (in-package :bld)
#<PACKAGE BLD>
BLD[4]> (setf *p* (program "test" '("test.c" "a.c" "b.c")))
#S(UNIT :NAME "test"
EPS
(#S(UNIT :NAME "test.o"
EPS (#S(UNIT :NAME "test.c" EPS NIL :TYPE
CFILE :CLEANABLE NIL))
:TYPE OFILE :CLEANABLE T)
#S(UNIT :NAME "a.o"
EPS (#S(UNIT :NAME "a.c" EPS NIL :TYPE CFILE :CLEANABLE
NIL))
:TYPE OFILE :CLEANABLE T)
#S(UNIT :NAME "b.o"
EPS (#S(UNIT :NAME "b.c" EPS NIL :TYPE CFILE :CLEANABLE
NIL))
:TYPE OFILE :CLEANABLE T))
:TYPE XFILE :CLEANABLE T)
BLD[5]> (build *p*)
[BLD] test.c: gcc -c -o test.o test.c
[BLD] a.c: gcc -c -o a.o a.c
[BLD] b.c: gcc -c -o b.o b.c
[BLD] test: gcc -o test test.o a.o b.o
NIL
BLD[6]> (clean *p*)
[CLN] rm test.o
[CLN] rm a.o
[CLN] rm b.o
[CLN] rm test
NIL
BLD[7]> (emit-to-screen nil)
NIL
BLD[8]> (emit-to-screen t)
T
BLD[9]>

Code is attached below for completeness. I will probably work
some more on this and continue to add more features in order to
learn lisp.

Parth

[My posts just seem to be getting longer and long ...
Sorry about that!]

;; ======================
;; String Utility Methods
;; ======================

(defun interleave (list &optional (sep " "))
"(interleave '('a' 'b' 'c' 'd') ' ') => ('a' ' ' 'b' ' ' 'c' ' '
'd')"
(loop for first = t then nil
for item in list
unless first collect sep
collect item))

(defun last-substr-pos-re (sub str)
"returns position of last sub-string.
e.g.: 'xx' 'abcxxyy' => 3"
(let ((m (all-matches sub str)))
(first (last (butlast m)))))

(defun replace-re (old new str &key (start 0))
"replace old with new in str. old and new are regular expressions"
; need to use concat as regex-replace
; drops begining when start is not 0
(concatenate 'string
(subseq str 0 start)
(regex-replace old str new :start start)))

(defun replace-last-re (old new str)
"replace the last occurance of old with new in str.
used to change file extension in strings.
e.g.: '\.c' '\.o' 'test.c' => 'test.o'"
(let ((pos (last-substr-pos-re old str)))
(when pos
(replace-re old new str :start pos))))

;; ========================
;; Checksum Utility Methods
;; ========================

; TODO: right now we just assume out of data tgt
; in case of file-error. ideally we should
; also check if the source exists
(defun target-out-of-date (src tgt)
(handler-case
(>= (file-write-date src) (file-write-date tgt))
(file-error () t)))

;;; ================================
;;; File Extension and Type Handling
;;; ================================
(defparameter *extension-map* '((".c" . cfile)
(".o" . ofile)
(".h" . hfile)))

(defun fext->symbol (ext)
"return the 'type' of file based on extension"
(rest (assoc ext *extension-map* :test #'equal)))

(defun file-ext (name)
(let ((dot-pos (search "." name :from-end t)))
(when dot-pos
(subseq name dot-pos))))

(defun fname->type (name)
(fext->symbol (file-ext name)))

; "test.c" => "test.o"
(defun cfile->ofile (name)
(replace-last-re "\.c" "\.o" name))

;;; ===================
;;; File Representation
;;; ===================

(defstruct (unit (:constructor %make-unit))
name
(deps nil :type list)
(type nil)
(cleanable nil))

(defun make-unit (&key name deps type (cleanable nil))
(%make-unit :name name
:deps deps
:type (or type (fname->type name))
:cleanable cleanable))

(defun make-deps-list (names)
(mapcar (lambda (x) (get-intermediate-unit (make-unit :name x)))
names))

(defun make-ofile (cfile)
(make-unit :name (cfile->ofile (unit-name cfile))
:deps (list cfile) :type 'ofile :cleanable t))

(defun get-intermediate-unit (file)
"takes a file and generates a plist for the possible intermediate
file once this is processed. For example, for a test.c plist
it will create a test.o plist. When there is no intermediate file,
input is returned"
(if (eq (fname->type (unit-name file)) 'cfile)
(make-ofile file)
file))

(defun program (target sources)
"top level program creation function. returns a tree that represents
files and their dependencies leading to the toplevel program."
(let ((tgt-unit (make-unit :name target)))
(setf (unit-deps tgt-unit) (make-deps-list sources))
(setf (unit-type tgt-unit) 'xfile)
(setf (unit-cleanable tgt-unit) t)
tgt-unit))

;;; ================
;;; Build Related
;;; ================

(defvar *ccflags* '("-o"))
(defvar *cc* "gcc")

;;; Top Level Build Method
;;; ----------------------
(defun build (program)
(if (null (unit-deps program))
(build-unit program)
(progn (mapcar #'build (unit-deps program))
(build-unit program))))

(defun build-unit (file)
"issues a build command"
(let ((cmd (funcall (file->build-cmd file) file)))
(when cmd
(format t "~&[BLD] ~A: " (unit-name file))
(funcall #'emit cmd))))

;;; Top Level Clean Method
;;; ----------------------
(defun clean (program)
(if (null (unit-deps program))
(clean-unit program)
(progn (mapcar #'clean (unit-deps program))
(clean-unit program))))

(defun clean-unit (program)
(when (unit-cleanable program)
(format t "[CLN] rm ~A~%" (unit-name program))
(when (not *emit-to-screen*)
(delete-file (unit-name program)))))

; deps-(<file0>, <file1>, <file2>) => ("depname0" "depname1"
"depname2")
(defun get-dep-names (file)
(mapcar (lambda (x) (unit-name x)) (unit-deps file)))

(defun build-cfile (file)
(let ((*ccflags* '("-c -o")))
`(,*cc* ,@*ccflags* ,(cfile->ofile (unit-name file)) ,(unit-name
file))))

(defun build-ofile (file) nil)

(defun build-xfile (file)
`(,*cc* ,@*ccflags* ,(unit-name file) ,@(get-dep-names file)))

(defparameter *build-cmd-map*
'((cfile . build-cfile)
(ofile . build-ofile)
(xfile . build-xfile)))

(defun file->build-cmd (f)
"return the 'type' of file based on extension"
(rest (assoc (unit-type f) *build-cmd-map* :test #'equal)))

(defun dump-program (p &optional (fname "tmp.dump"))
(with-open-file (out fname :direction utput :if-exists :supersede)
(write p :stream out)))

;; when set to t, emitted commands are executed
(defparameter *emit-to-screen* t)

(defun emit (cmd)
(if *emit-to-screen*
(format t "~A~%" (lst->str cmd))
(execute-shell-command-to-stdout (lst->str cmd))))

(defun emit-to-screen (&optional (to-screen t))
(setf *emit-to-screen* to-screen))

(defun lst->str (lst)
(format nil "~{~A~^ ~}" lst))

;;; ==================
;;; System Interaction
;;; ==================

; TODO: make this portable across implementations.
; atleast throw errors on implementations other
; than clisp.
(defun execute-shell-command (cmd)
(ext:make-pipe-input-stream cmd))

(defun execute-shell-command-to-stdout (cmd)
(format t "~A~%" cmd)
(when cmd
(let ((in (execute-shell-command cmd)))
(when in
(loop for line = (read-line in nil)
while line do (format t "~A~%" line))
(close in)))))

Den Sat, 14 Jun 2008 19:06:24 -0700 skrev parth.malwankar:
> For now I have upgraded to using structs. As Sasha suggested I would
> probably upgrade to classes as I add this app evolves some more.

Note that you can dispatch methods just fine on structs, they are types
too.

Cheers,
Maciej

> BLD[7]> (emit-to-screen nil)
> NIL
> BLD[8]> (emit-to-screen t)
> T

This is a bit weird. On reading this I'd think that EMIT-TO-SCREEN,
will, uh, emit some thing to some screen. But it doesn't, it's just
a switch.

Functional interfaces can get very clumsy when
you need to keep a bunch of states.

EMIT-TO-SCREEN should be SETFable.

What I'd do:

(defstruct program
... emit-to-screen)

(defmethod build ((program program))
...)

To be used thus:

(let ((program (make-program :emit-to-screen t))
(setf (program-emit-to-screen) nil)
(build program))

Better yet:

(defstruct program
... emit-to)

(setf (program-emit-to) :screen)

On Jun 16, 1:03 pm, "Leslie P. Polzer" <leslie.pol...@gmx.net> wrote:
> > BLD[7]> (emit-to-screen nil)
> > NIL
> > BLD[8]> (emit-to-screen t)
> > T
>
> This is a bit weird. On reading this I'd think that EMIT-TO-SCREEN,
> will, uh, emit some thing to some screen. But it doesn't, it's just
> a switch.
>

I suppose the interaction sample I provied is a little confusion.
I was thinking of emit-to-screen as just a helper function. It
provide a convenient way of setting/resetting the *emit-to-screen*
global. The subsequent build / clean would either actually do
something or just print to screen based on this flag.

;; when set to t, emitted commands are executed
(defparameter *emit-to-screen* t)

(defun emit-to-screen (&optional (to-screen t))
(setf *emit-to-screen* to-screen))

(defun emit (cmd)
(if *emit-to-screen*
(format t "~A~%" (lst->str cmd))
(execute-shell-command-to-stdout (lst->str cmd))))

The 'emit' function takes this into account and either just
prints the commands to screen or executes them. I was thinking
of this more like the '-n' option in gmake.


> Functional interfaces can get very clumsy when
> you need to keep a bunch of states.
>

True.

> EMIT-TO-SCREEN should be SETFable.
>
> What I'd do:
>
> (defstruct program
> ... emit-to-screen)
>
> (defmethod build ((program program))
> ...)
>
> To be used thus:
>
> (let ((program (make-program :emit-to-screen t))
> (setf (program-emit-to-screen) nil)
> (build program))
>
> Better yet:
>
> (defstruct program
> ... emit-to)
>
> (setf (program-emit-to) :screen)

This sounds like actually quite a good approach.
Along the same lines, rather than a separate
function *emit-to-screen* maintaining state, this could
be an optional parameter to 'build' and 'clean' (being
subsequently passed to emit.

This way when I add support for command line options, a
-n on the command line could correspond to t being
passed to :to-screen for build / clean as the case may
be.

Thanks.

On Jun 16, 1:03 pm, "Leslie P. Polzer" <leslie.pol...@gmx.net> wrote:
> > BLD[7]> (emit-to-screen nil)
> > NIL
> > BLD[8]> (emit-to-screen t)
> > T
>
> This is a bit weird. On reading this I'd think that EMIT-TO-SCREEN,
> will, uh, emit some thing to some screen. But it doesn't, it's just
> a switch.
>
> Functional interfaces can get very clumsy when
> you need to keep a bunch of states.
>
> EMIT-TO-SCREEN should be SETFable.
>
> What I'd do:
>
> (defstruct program
> ... emit-to-screen)
>
> (defmethod build ((program program))
> ...)
>
> To be used thus:
>
> (let ((program (make-program :emit-to-screen t))
> (setf (program-emit-to-screen) nil)
> (build program))
>
> Better yet:
>
> (defstruct program
> ... emit-to)
>
> (setf (program-emit-to) :screen)

This makes sense. I have pulled *emit-to-screen* along
with *ccflags* and *cc* into a build-env struct.

Two possible interactions:

;; Usage 0
;; =======
; (setf *p* (program "test" '("test.c" "a.c" "b.c")))
; (build *p*)
; (clean *p*)
;
;; Usage 1
;; =======
; (setf *env* (make-build-env :emit-to-screen nil))
; (setf *p* (program "test" '("test.c" "a.c" "b.c")))
; (build *p* *env*)
; (clean *p* *env*)

Basically the build-env now defaults to gcc and emits
to screen. The user has a choice to create another env
and pass it to build / clean. The typical usage would
be have multiple build environments like debug / release.

(defstruct build-env
(ccflags '("-o"))
(cc "gcc")
(program nil)
(emit-to-screen t))

(defparameter *build-environment* (make-build-env))


I have hosted the full code at google code to avoid cluttering
the newsgroup.
http://code.google.com/p/bld/source/...k/src/bld.lisp

Thanks very much for your suggestions.
Parth