request comments: simple c program builder in lisp

parth.malwankar@gmail.com writes:


Since a lot of commentary has already occurred, I'll just pick and
choose a few items (some of which are a bit redundant).

> Source Code Listing
> ===================

> (defun concat-strings-from-list (l)
> "take all strings in the list and concatenate them
> adding a space in between.
> e.g.: ('aa' 'bb' 'cc') => 'aa bb cc'"
> (reduce #'(lambda (x y) (concatenate 'string x " " y))
> l))

There is also the FORMAT option, already mentioned:

(format nil "~{~A~^ ~}" list)

This has the additional advantage that it will work with lists of
arbitrary objects and not just strings. So symbols, numbers, etc will
also work easily.

If you want a more general purpose function, you could write something
like the following, which is actually a bit simpler to write using the
reduce method:

(defun format-list (list &optional (delimiter " "))
"Formats LIST items separated by DELIMITER"
(reduce #'(lambda (x y) (concatenate 'string x delimiter y))
list :initial-value ""))

This then can be used more generally. Also, if you add the
:INITIAL-VALUE keyword, then the function will work correctly even if
you pass in NIL as the list to concatenate. It will then return just
the empty string instead of signaling an error.

To do the same thing in format is a bit uglier, since you would need to
construct the format string as part of applying the function:

... (format (concatenate 'string "~{~A~^" delimiter "~}") ...)

> (defun find-last-subst-pos-re (sub str)
> "returns position of last sub-string.
> e.g.: 'xx' 'abcxxyy' => 3"
> (let ((m (all-matches sub str)))
> (first (last (butlast m)))))

If this is truly just looking for the substring rather than the
occurrence of of a regular expression, there is a more straightforward
method using built-in Common Lisp forms. And if you really mean it to
be regular expressions, the documentation string should say that rather
than "sub-string".

(search sub str :from-end t)

> ;;; ================================
> ;;; File Extension and Type Handling
> ;;; ================================
>
> (defun extention-p (ext name)
> "return if name ends in ext. used to check file extension"
> (let ((ext-len (length ext)))
> (equal ext (subseq name (- (length name) ext-len)))))

Should be called something like HAS-EXTENSION or EXTENSION-MATCHES.

I would also write another helper function called GET-EXTENSION that
attempts to just return the extension of a file. This would typically
just look for the last "." in the name and return whatever starts there:

(defun get-extension (name)
"Returns the extension part of "name".
(let ((extension-start (position #/. name :from-end t)))
(when extension-start
(subseq name extension-start))))

Or even better, you could use the Common Lisp pathname functions and use
PATHNAME-TYPE to get the extension. That will also insulate more of
your code from the underlying operating system's syntax for indicating
file types. Then you would simply be able to write

(defun has-extension (name ext)
(string= ext (pathname-type name)))

Note that you would then have to use "c" rather than ".c" as the
extension name. But this is a lot simpler.


> (defun filetype-p (name type)
> "check if file is of a 'type'"
> (cond ((equal type 'cfile) (extention-p ".c" name))
> ((equal type 'ofile) (extention-p ".o" name))
> ((equal type 'hfile) (extention-p ".h" name))
> (t nil)))
>
> (defun filetype? (name)
> "return the 'type' of file based on extension"
> (cond ((extention-p ".c" name) 'cfile)
> ((extention-p ".o" name) 'ofile)
> ((extention-p ".h" name) 'hfile)
> (t nil)))

As has been noted, this would be better handled by using a declarative
data structure such as an association list to store the correspondences:

(defparameter *extension-mapping*
'(("c" . cfile) ("o" . ofile) ("h" . hfile)))

This then lets you write some very simple functions to get items out of
the data structure. By using this declarative approach, you are also
assured that any changes to make will always be reflected in the
underlying code, since you won't have to remember to update both
functions.

(defun get-file-type (name)
(cdr (assoc (pathname-type name) *extension-mapping*
:test #'string=)))

(defun get-file-extension (type)
(car (rassoc type *extension-mapping*)))

(defun has-file-type (name type)
(eql (get-file-type name) type))

The general theme is to break a lot of this down to very simple
functions that are generally applicable and that also build on one
another.

> ;;; ===================
> ;;; File Representation
> ;;; ===================

If you change the direct file representation to be pathnames instead of
just strings, then some of these manipulations would be easier.

> (defun make-file (&key name (deps nil) (type nil))
> (list :name name :deps deps :type (filetype? name)))

I would use this to convert NAME to a pathname object with
(pathname name)

> (defun make-file-list (names)
> (mapcar #'(lambda (x) (get-intermediate-file (make-file :name x)))
> names))
>
> (defun get-intermediate-file (file)
> "takes a file and generates a plist for the possible intermediate
> file once this is processed. For example, for a test.c plist
> it will create a test.o plist. When there is no intermediate file,
> input is returned"
> (let ((name (getf file :name))
> (type (getf file :type)))
> (cond ((equal type 'cfile) (make-file
> :name (search-replace-last-re "\.c"
> "\.o" name)
> :deps (list file)
> :type 'ofile))
> (t file))))

Here you might also want to build a data structure similar to the one
used to map file extensions to types and use that to contain the mapping
from files to successor types. That then makes the function easily
extensible and keeps the correspondence between successors where you can
easily see it, instead of buried inside the code for this function.

Note that this assume you convert file names to pathnames.

(defparameter *file-successors* '((cfile . ofile)))

(defun change-file-type (name new-type)
(let ((new-name (pathname (namestring name))))
(setf (pathname-type new-name) (get-file-extension new-type))
new-name))

(defun get-successor-file (file)
(let ((name (getf file :name))
(next-type (cdr (assoc (getf file :type) *file-successors*))))
(if next-type
(make-file :name (change-file-type name next-type)
:deps (list file)
:type next-type)
file)))

>
> ; Usage:
> ; (setf *p* (program "test" '("a.c" "b.c" "c.d")))
> ; (build *p*)
>
> (defun program (target sources)
> "top level program creation function. returns a tree that represents
> files and their dependencies leading to the toplevel program."
> (let ((target-file (make-file :name target)))
> (setf (getf target-file :deps) (make-file-list sources))
> (setf (getf target-file :type) 'exe)
> target-file))
>
> (defun build (program)
> (let ((deps (getf program :deps))
> (name (getf program :name))
> (type (getf program :type)))
> (cond ((null deps) (format t "[build] ~A: " name)
> (build-file program))
> (t (mapcar #'build deps)
> (format t "[build] ~A: " name)
> (build-file program)))))
>
> ; "test.c" => "test.o"
> (defun cfile->ofile (name)
> (search-replace-last-re "\.c" "\.o" name))
>
> ; deps-(<file0>, <file1>, <file2>) => ("depname0" "depname1"
> "depname2")
> (defun get-dep-names (file)
> (let ((deps (getf file :deps)))
> (mapcar #'(lambda (x) (getf x :name)) deps)))
>
> (defun build-file (file)
> "print the string needed to build the file based on file type"
> (let ((type (getf file :type))
> (name (getf file :name))
> (deps (getf file :deps)))
> (cond ((equal type 'cfile)
> (format t "~A~%" (concat-strings-from-list
> (list "gcc -o" (cfile->ofile name)
> name))))
> ((equal type 'ofile)
> (format t "nothing to do: ~A~%" name))
> ((equal type 'exe)
> (format t "~A~%" (concatenate 'string
> "gcc -o "
> name " "
> (concat-strings-from-list
> (get-dep-names file)))))
> (t (format t "nothing to do: ~A~%" name)))))
>

--
Thomas A. Russ, USC/Information Sciences Institute

On Jun 17, 12:22 am, t...@sevak.isi.edu (Thomas A. Russ) wrote:
> parth.malwan...@gmail.com writes:
>
> Since a lot of commentary has already occurred, I'll just pick and
> choose a few items (some of which are a bit redundant).
>
> > Source Code Listing
> > ===================
> > (defun concat-strings-from-list (l)
> > "take all strings in the list and concatenate them
> > adding a space in between.
> > e.g.: ('aa' 'bb' 'cc') => 'aa bb cc'"
> > (reduce #'(lambda (x y) (concatenate 'string x " " y))
> > l))
>
> There is also the FORMAT option, already mentioned:
>
> (format nil "~{~A~^ ~}" list)
>
> This has the additional advantage that it will work with lists of
> arbitrary objects and not just strings. So symbols, numbers, etc will
> also work easily.
>
> If you want a more general purpose function, you could write something
> like the following, which is actually a bit simpler to write using the
> reduce method:
>
> (defun format-list (list &optional (delimiter " "))
> "Formats LIST items separated by DELIMITER"
> (reduce #'(lambda (x y) (concatenate 'string x delimiter y))
> list :initial-value ""))
>
> This then can be used more generally. Also, if you add the
> :INITIAL-VALUE keyword, then the function will work correctly even if
> you pass in NIL as the list to concatenate. It will then return just
> the empty string instead of signaling an error.
>
> To do the same thing in format is a bit uglier, since you would need to
> construct the format string as part of applying the function:
>
> ... (format (concatenate 'string "~{~A~^" delimiter "~}") ...)
>
> > (defun find-last-subst-pos-re (sub str)
> > "returns position of last sub-string.
> > e.g.: 'xx' 'abcxxyy' => 3"
> > (let ((m (all-matches sub str)))
> > (first (last (butlast m)))))
>
> If this is truly just looking for the substring rather than the
> occurrence of of a regular expression, there is a more straightforward
> method using built-in Common Lisp forms. And if you really mean it to
> be regular expressions, the documentation string should say that rather
> than "sub-string".
>
> (search sub str :from-end t)
>
> > ;;; ================================
> > ;;; File Extension and Type Handling
> > ;;; ================================
>
> > (defun extention-p (ext name)
> > "return if name ends in ext. used to check file extension"
> > (let ((ext-len (length ext)))
> > (equal ext (subseq name (- (length name) ext-len)))))
>
> Should be called something like HAS-EXTENSION or EXTENSION-MATCHES.
>
> I would also write another helper function called GET-EXTENSION that
> attempts to just return the extension of a file. This would typically
> just look for the last "." in the name and return whatever starts there:
>
> (defun get-extension (name)
> "Returns the extension part of "name".
> (let ((extension-start (position #/. name :from-end t)))
> (when extension-start
> (subseq name extension-start))))
>
> Or even better, you could use the Common Lisp pathname functions and use
> PATHNAME-TYPE to get the extension. That will also insulate more of
> your code from the underlying operating system's syntax for indicating
> file types. Then you would simply be able to write
>
> (defun has-extension (name ext)
> (string= ext (pathname-type name)))
>
> Note that you would then have to use "c" rather than ".c" as the
> extension name. But this is a lot simpler.
>
> > (defun filetype-p (name type)
> > "check if file is of a 'type'"
> > (cond ((equal type 'cfile) (extention-p ".c" name))
> > ((equal type 'ofile) (extention-p ".o" name))
> > ((equal type 'hfile) (extention-p ".h" name))
> > (t nil)))
>
> > (defun filetype? (name)
> > "return the 'type' of file based on extension"
> > (cond ((extention-p ".c" name) 'cfile)
> > ((extention-p ".o" name) 'ofile)
> > ((extention-p ".h" name) 'hfile)
> > (t nil)))
>
> As has been noted, this would be better handled by using a declarative
> data structure such as an association list to store the correspondences:
>
> (defparameter *extension-mapping*
> '(("c" . cfile) ("o" . ofile) ("h" . hfile)))
>
> This then lets you write some very simple functions to get items out of
> the data structure. By using this declarative approach, you are also
> assured that any changes to make will always be reflected in the
> underlying code, since you won't have to remember to update both
> functions.
>
> (defun get-file-type (name)
> (cdr (assoc (pathname-type name) *extension-mapping*
> :test #'string=)))
>
> (defun get-file-extension (type)
> (car (rassoc type *extension-mapping*)))
>
> (defun has-file-type (name type)
> (eql (get-file-type name) type))
>
> The general theme is to break a lot of this down to very simple
> functions that are generally applicable and that also build on one
> another.
>
> > ;;; ===================
> > ;;; File Representation
> > ;;; ===================
>
> If you change the direct file representation to be pathnames instead of
> just strings, then some of these manipulations would be easier.
>
> > (defun make-file (&key name (deps nil) (type nil))
> > (list :name name :deps deps :type (filetype? name)))
>
> I would use this to convert NAME to a pathname object with
> (pathname name)
>
>
>
> > (defun make-file-list (names)
> > (mapcar #'(lambda (x) (get-intermediate-file (make-file :name x)))
> > names))
>
> > (defun get-intermediate-file (file)
> > "takes a file and generates a plist for the possible intermediate
> > file once this is processed. For example, for a test.c plist
> > it will create a test.o plist. When there is no intermediate file,
> > input is returned"
> > (let ((name (getf file :name))
> > (type (getf file :type)))
> > (cond ((equal type 'cfile) (make-file
> > :name (search-replace-last-re "\.c"
> > "\.o" name)
> > :deps (list file)
> > :type 'ofile))
> > (t file))))
>
> Here you might also want to build a data structure similar to the one
> used to map file extensions to types and use that to contain the mapping
> from files to successor types. That then makes the function easily
> extensible and keeps the correspondence between successors where you can
> easily see it, instead of buried inside the code for this function.
>
> Note that this assume you convert file names to pathnames.
>
> (defparameter *file-successors* '((cfile . ofile)))
>
> (defun change-file-type (name new-type)
> (let ((new-name (pathname (namestring name))))
> (setf (pathname-type new-name) (get-file-extension new-type))
> new-name))
>
> (defun get-successor-file (file)
> (let ((name (getf file :name))
> (next-type (cdr (assoc (getf file :type) *file-successors*))))
> (if next-type
> (make-file :name (change-file-type name next-type)
> :deps (list file)
> :type next-type)
> file)))
>
>

Thanks Thomas. Coming from other languages pathnames doesn't come
naturally to be (yet!) but this makes a lot of sense.

I have updated the code to use pathnames and also generalized
using *file-successors*.
Was able to get rid of regex as this was being used only for
file extension which is easier and more reliable with pathname.

Parth

>
>
>
> > ; Usage:
> > ; (setf *p* (program "test" '("a.c" "b.c" "c.d")))
> > ; (build *p*)
>
> > (defun program (target sources)
> > "top level program creation function. returns a tree that represents
> > files and their dependencies leading to the toplevel program."
> > (let ((target-file (make-file :name target)))
> > (setf (getf target-file :deps) (make-file-list sources))
> > (setf (getf target-file :type) 'exe)
> > target-file))
>
> > (defun build (program)
> > (let ((deps (getf program :deps))
> > (name (getf program :name))
> > (type (getf program :type)))
> > (cond ((null deps) (format t "[build] ~A: " name)
> > (build-file program))
> > (t (mapcar #'build deps)
> > (format t "[build] ~A: " name)
> > (build-file program)))))
>
> > ; "test.c" => "test.o"
> > (defun cfile->ofile (name)
> > (search-replace-last-re "\.c" "\.o" name))
>
> > ; deps-(<file0>, <file1>, <file2>) => ("depname0" "depname1"
> > "depname2")
> > (defun get-dep-names (file)
> > (let ((deps (getf file :deps)))
> > (mapcar #'(lambda (x) (getf x :name)) deps)))
>
> > (defun build-file (file)
> > "print the string needed to build the file based on file type"
> > (let ((type (getf file :type))
> > (name (getf file :name))
> > (deps (getf file :deps)))
> > (cond ((equal type 'cfile)
> > (format t "~A~%" (concat-strings-from-list
> > (list "gcc -o" (cfile->ofile name)
> > name))))
> > ((equal type 'ofile)
> > (format t "nothing to do: ~A~%" name))
> > ((equal type 'exe)
> > (format t "~A~%" (concatenate 'string
> > "gcc -o "
> > name " "
> > (concat-strings-from-list
> > (get-dep-names file)))))
> > (t (format t "nothing to do: ~A~%" name)))))
>
> --
> Thomas A. Russ, USC/Information Sciences Institute