*Huge* text file import

This is a discussion on *Huge* text file import within the basic.visual forums in Programming Languages category; I have written a program that performs a nightly text import from files with fixed width fields in the rows. Each file has a fixed line length. The files are so large that it's taking hours to read the files and load them into the database. I am using this code: <pre> Public Const MAX_ARRAY_LOOP = 50000 Public fileLineNumber As Long Public recordArray() As String Public recordArraySize As Long Private Sub loadRecordArray(sFilename As String) Dim fileLineSize As Long Dim i As Long If recordArraySize > MAX_ARRAY_LOOP Then fileLineNumber = 1 fileLineSize = recordArraySize recordArraySize = 1 Open sFilename For Binary ...

Go Back   Application Development Forum > Programming Languages > basic.visual

Object Mix

Register FAQ Calendar Search Today's Posts Mark Forums Read
  #1  
Old 09-03-2008, 01:15 PM
teresa
Guest
 
Default *Huge* text file import

I have written a program that performs a nightly text import from
files with fixed width fields in the rows. Each file has a fixed line
length. The files are so large that it's taking hours to read the
files and load them into the database.

I am using this code:
<pre>

Public Const MAX_ARRAY_LOOP = 50000
Public fileLineNumber As Long
Public recordArray() As String
Public recordArraySize As Long

Private Sub loadRecordArray(sFilename As String)
Dim fileLineSize As Long
Dim i As Long

If recordArraySize > MAX_ARRAY_LOOP Then
fileLineNumber = 1
fileLineSize = recordArraySize
recordArraySize = 1
Open sFilename For Binary As #fn
Do
For i = 1 To MAX_ARRAY_LOOP
If i > fileLineSize Then
Exit For
End If
ReDim Preserve recordArray(recordArraySize)
Get #fn, , varString
recordArray(recordArraySize) = varString
If i <> MAX_ARRAY_LOOP Then
recordArraySize = recordArraySize + 1
fileLineNumber = fileLineNumber + 1
End If
Next i
'sub that inserts recordArray into database
insertRecords
recordArraySize = 1
ReDim recordArray(0)
Loop Until EOF(fn)
Close #fn
Else
Open sFilename For Binary As #fn
recordArraySize = 0
Do
recordArraySize = recordArraySize + 1
Get #fn, , varString
If Mid(varString, 1, 1) <> Chr(0) Then
ReDim Preserve recordArray(recordArraySize)
recordArray(recordArraySize) = varString
Else
'error record
End If
Loop Until EOF(fn)
'sub that inserts recordArray into database
insertRecords
Close #fn
End If
End Sub

</pre>

The files I'm importing today are 697,999KB, 367,422KB, and
210,897KB. I really need something faster than what I've got; does
anyone have an idea that I can use?

Thanks,
Teresa
Reply With Quote
Reply


Thread Tools
Display Modes


All times are GMT -5. The time now is 02:56 AM.


Powered by vBulletin® Version 3.7.2
Copyright ©2000 - 2008, Jelsoft Enterprises Ltd.
Search Engine Optimization by vBSEO 3.2.0
vB Ad Management by =RedTyger=

In an effort to better serve ads to our visitors, cookies are used on objectmix.com. For more information, check out our Privacy Policy.