Large text file

Trying to put a file into a variable and it’s just hanging. Only difference I can tell from any other file I have been trying to put into a variable is the size. It’s a 3.3mb txt file. When I put it into the var, it just sits there and hangs, never to continue. This is a problem. I need to know how to put large files sizes into variables. I’ve restarted eggplant several times, rebooted the MAC, still same result. It just hangs. Any help?

This may have something to do with the contents of the file, or perhaps the way the file is encoded. I just loaded a 4.5 MB file into a variable and then did a put on the variable and the script ran in 3 seconds. So the size of the file would not appear to be the problem.

Hmmmm…

is there a way to encode it or format it via sensetalk?

I’ve tried opening the file and resaving it on the Mac. It’s being saved as Western (Mac OS Roman) - Plain Text Encoding.

Have you tried setting the defaultStringEncoding to match what the file is being saved as?

set the defaultStringEncoding to "Western (Mac OS Roman)"

Ok, so the file contains 176,000 lines…I’ve kinda gotten around it by using the open file method and a repeat, but I’m still gonna have a killer time reading everything, especially if the data I need is at the bottom of the file, which it is. Any ideas?

Can you send the file to support and we can have a look at it and figure out why you’re having this problem? You should be able to use your approach and use “read … until end” to read the whole file into a variable. Is there any chance that your file is open and being written to by another process? It almost seems like eggPlant isn’t recognizing the end of the file as the being the end.

Well I can’t just quite do that yet. The file contains information which is private and would be against policy to send. I could however send a file that is the same size and which the same amount of lines.

It seems to be when I “log” it. If I “put” it. It takes no time and functions as intended. If I “log” it…well I get nothing. “Line” by “Line” is fine, but if I need something at the bottom of the file, which I do. It would be a nightmare waiting for something for 5 minutes. Logging everything gets me just a blank stare from my Mac.

I just sent a file to support that has the same results that I recreated with the same amount of lines and the same exact file size.

I turned trace on and this is what I see…

<<< 01:25:10 End handler 2: getsymbolpositions (aborted) —my function
<<< 01:25:10 End handler 1: paylineandsymboltemplate (aborted) —my script

but it never aborted the script, it’s still “running”. Clicking “abort” does nothing, I have to force quit the program to get any functionality of eggplant back.

Are you trying to log the whole file? That’s going to be huge mess. It goes into the results for the script in field that’s intended for short messages. And you’re writing 3.5 MB into the file – that’s not something we expect people to do, so I doubt there’s any optimization around that action.

We can make a file that size – I already tested with a larger one – so sending us a file with the same number of lines isn’t necessary.

“Line” by “Line” is fine, but if I need something at the bottom of the file, which I do. It would be a nightmare waiting for something for 5 minutes.

Can you describe exactly what you’re trying to do? Did you try using “read … until end” to put the whole thing into a variable at once?

Okay, it looks like you’ve discovered a severe performance problem. It turns out it’s not a problem with reading the file (which is very fast) but with counting or accessing the lines of text. The problem is being fixed for the next release – thanks for pointing it out!

In the meantime, you can work around it very easily by changing the lineDelimiter before working with the lines of text:

set the lineDelimiter to return

That should eliminate the problem for now.